Why maintainers become frustrated

As a maintainer, my first reaction to a pull request from a newcomer used to be excitement - open source is a social experience and helping someone get involved is really rewarding - I love helping people take their first steps into open source, and who knows, they might even become a regular contributor.

Things have changed a lot in the past year or two; now the first thing I do is try to work out “has this person used AI and does it look like they read the code before submitting this PR?”

It’s frustrating. I don’t mind reviewing code which has been generated by AI but then fully understood by a human who is able to make any requested changes, explain the reasoning behind the design choices, and have ownership of the code even if they didn’t write it themselves.

But when the author hasn’t engaged with the code, the review process can be a real drag. Some authors abandon their PR upon receiving any feedback, others respond with more AI and no real human engagement. In these cases, I’d rather save the time and effort, and just make the change using AI myself - I’ll certainly do a better job. It’s also a chunk of emotional energy - I want to be welcoming and friendly, and if I come across a heavily AI-generated PR, there’s still the chance that the author may be willing to engage during the review process, and so I don’t want to say “no” immediately or reply grumpily, as I still want to be part of creating a welcoming environment.

I don’t have a good solution here other than to give these PRs as quick an initial review as possible to gauge the vibes of the author based on their next response. Some projects I’m involved with have turned on automatic AI code reviews for new contributors, to deal with the sheer volume of new contributions.

I’m not always in the maintainer role though, and when dealing with author people’s codebases, I’m now the external contributor using AI to generate code to submit PR. The rest of this post covers the steps I take to try to not frustrate other maintainers.

Step 1 - Build trust gradually

Code contributions can feel satisfying but aren’t always the most helpful, and there are so many other ways to contribute to a codebase. I don’t always make PRs. I know they’re a time commitment - after generating the code, I also need to stick around for the iteration of review and updates and I don’t always have time for this. If I find a bug, I’ll often just report it, with a simple reproducible example. I might also take the time to work out with AI the source of the bug, and if I’m confident about the explanation, include that in the issue too. If the fix is simple, I’ll submit a PR, but if there are complex design decisions to be made, I’ll often get package author input.

Of course, not all contributions have to be code, and a great non-code contribution is triaging other people’s bugs, by adding simple reproducible examples if they’re missing.Or explanations of the cause if I know it, to make it easier for maintainers to get it fixed faster.

I also try to stick to the principle of starting small and building up trust. I’ll submit a PR for a documentation fix if I spot a typo, or fix an existing bug. I rarely submit PRs for feature implementations, unless I’ve been contributing for a while or have buy-in from the package maintainers by opening an issue and asking if they’d be happy for me to make a PR.

Generally, I want to demonstrate that I’ve read any contribution guidelines, can submit code with the correct level of testing and in the style of the rest of the codebase. And on that note…

Step 2 - Learning the codebase/fitting it

The single biggest piece of advice I follow is “learn to fit in”. Essentially, what I’m aiming for is that my PR looks like other PRs which have been merged. This extends to the smallest details - even if it means using `=` instead of `<-`, you’ve got leave your principles at the door, or run the risk of doing the equivalent of going into someone else’s house and rearranging their furniture!

I like to think of it as if I’m some sort of spy or undercover agent who is trying to slide into their ecosystem so smoothly that my PR barely feels external. My favourite word I use to describe what I’m trying to be here is “idiomatic”. There are often multiple ways to write “good” or “correct” code, so which particular way of doing that is being used most in this codebase?

So, how do I do that? A few things:

• look at past PRs from maintainers to see if they have linked issues, how much testing is done, what the PR descriptions look like and how much detail is in there
• do the same but with PRs from non-maintainers to see where gets pushed back on most for additional details etc
• skim the unit tests; what is tested and what isn’t?
• skim the code; how many comments are there, how modular is the code, what conventions seem to be used (e.g. lapply() vs. purrr::map(); how is code split out into files etc?)
• which previous PRs can I use as a “template” for what to do here?

Step 3 - Writing the code

Now I know how to blend in, I can write some code. The approach I take varies depending on how much I understand the underlying code and what I’m doing.

If it’s a bug fix, I tend to generate it entirely with AI. I use Claude and whatever the latest model is with reasoning set to at least medium. I’ll let Claude work through some root cause analysis, and repeatedly question it on its assumptions. If I don’t know enough to verify the conclusion myself, I’ll sometimes ask it to try to prove the opposite of its conclusion, or explore more alternative hypotheses. Once I’m satisfied with the conclusions, I’ll have it write the fix, usually questioning it about alternative approaches and design choices.

Feature implementation is a whole lot trickier, and I actually use a Claude skill I wrote which designs “rapid learning” sessions for me. I only use this when I’m so unfamiliar with the codebase that I wouldn’t feel able to review the produced code effectively. Essentially, I have Claude spend time looking at the problem, and then walking me through understanding the relevant code areas so I’m able to fully understand the context. We’ll set up a concrete goal, set the time frame for the learning, and before Claude implements the fix, it’ll have me read certain subsections of code and then ask me what needs to change and where and why, and it’s only at that point I actually use it to make the change.

Doing this means I’m able to fully take ownership of the code and review it before I submit the pull request.

Step 4 - Reviewing the code

This is the most essential stage and is where it’s easy to suddenly realise that the skillset needed to contribute code to open source hasn’t shrunk, it’s just changed.

Previously my code contributions were bottom-up and I’d work through a problem with step-by-step adding code but now it’s more the case of having a finished solution and working backwards to figure out why and how and what.

If I’ve been using my “rapid learning” Claude skill, I’ve already got the reasoning in my head, but if not then it’s a case of reading through the code line by line and looking at all of the changes just to try to understand why this is the right change. I might ask the AI agent about the reasoning, but this is a point where I also look for analogous similar code or think about the different ways that I could have implemented this change. I’ll make sure that the comments are only on none-obvious changes and the testing matches the rest of the code base. This is where the work that I did in step 2 becomes useful, because to successfully review the code, I have to know what I’m aiming for here.

If the generated code is complex and I’ve spent a while iterating on a solution, I’m pretty awful at giving it an honest review immediately after generating it, so at this point I might leave it a few days before coming back to it to have a bit of distance from it and so I read it properly rather than skim-reading it.

Reviewing is the hardest task of the lot, and it’s the kind of thing that I’ll do at the start of the day when I’ve got the most energy. If I don’t do this, I find myself missing small details really easily.

Step 5 - PR description

The last stage is submitting the actual pull request. I’ll often try to include a description of the changes that I’ve made, again, as per other pull requests in the repository. It’s so important here not to just add in a ton of extra detail that isn’t needed though, which seems to be the default LLM behaviour here. Worse, LLMs often generate inaccurate PR descriptions which don’t represent the final state of the PR after changes have been made.

As a reviewer I find lengthy descriptions really add to my cognitive load, and they put me off even attempting to review the PR.

If there were any decisions made by AI that I’m not sure about myself, at this point I will explicitly call them out by adding review comments in the GitHub UI to my own pull requests. This reduces the chance of subtle errors slipping past maintainers but it also signals my willingness to engage with the review process and the fact that I’ve engaged with the code.

Conclusion

It’s not perfect, and I’m still working on these skills, but I believe that doing the above has enabled me to make PRs to open source projects that have helped me both grow as a developer but also given something useful to the project. The core thread running through all of this is keeping the human firmly in the loop.

It reminds me a bit of visiting Toulouse in 2019 for useR! conference. There’s a stereotype I’ve heard repeatedly throughout my life about shopkeepers being rude in France, and I’d attributed it to overgeneralisations stemming from the historical rivalry between England and France, and so I tried to not pay it too much attention. But when I was there, I really did notice some weird vibes when out and about. I was a bit irritated - I can be a little sensitive sometimes and I always believed I was being polite with every request in my broken French having a “s’il vous plait” at the end.

But later during that trip, I learned that going into a shop and launching straight into a request without so much as a “bonjour” is widely considered kinda rude, and once I started making an effort to say hello first, those weird vibes mostly disappeared.

When I think about it, I actually quite like that social rule and think it applies much more broadly in life whether travelling internationally or working on open source - engage as a human, and you’ll get much better results.

Huge thanks to Alenka Frim and Josiah Parry for giving an earlier draft of this a read over!

What is Parquet and why should you care?

• Smaller than equivalent CSV files so you can work with and share bigger datasets more easily
• Way faster to work with than CSVs
• Stores data types so less room for error or custom code needed to convert things when you read it in

How?

• Smaller than equivalent CSV - internally uses encoding and compression and is a binary format (made for computers not humans)
• Faster - data stored in small pieces and in columns so can read efficiently and operate in parallel
• Data types - Parquet files store metadata internally about the data itself but also and how it’s been saved

We’ll look at these more closely later!

What packages do I need to work with Parquet files?

Pick one of…

• arrow - for all features, including working with multi-file datasets
• nanoparquet - for a no-dependency package which has most (but not all) features but can be slower on larger datasets
• duckdb - for SQL-oriented workflows

Today we’ll focus on {arrow} but some examples with {nanoparquet}

How do you open a Parquet file?

You can use read_parquet() from arrow (or nanoarrow!)

library(arrow)

Attaching package: 'arrow'

The following object is masked from 'package:utils':

    timestamp

library(tibble)

taxi <- read_parquet(
  "https://arrow-datasets.s3.amazonaws.com/nyc-taxi-tiny/year=2019/month=1/part-0.parquet"
)
# Below is the same file but locally
taxi <- read_parquet("taxi-mini.parquet")

# it's read in as a tibble automatically
class(taxi)

[1] "tbl_df"     "tbl"        "data.frame"

taxi

# A tibble: 7,667 × 22
   vendor_name pickup_datetime     dropoff_datetime    passenger_count
   <chr>       <dttm>              <dttm>                        <int>
 1 CMT         2019-01-25 22:15:29 2019-01-25 23:03:20               2
 2 CMT         2019-01-18 14:39:02 2019-01-18 14:45:36               1
 3 VTS         2019-01-24 15:02:06 2019-01-24 15:24:38               1
 4 VTS         2019-01-01 11:23:33 2019-01-01 12:04:09               1
 5 CMT         2019-01-14 13:33:39 2019-01-14 13:46:13               2
 6 CMT         2019-01-26 19:26:48 2019-01-26 19:31:46               1
 7 VTS         2019-01-15 10:03:54 2019-01-15 10:38:38               1
 8 CMT         2019-01-26 10:17:39 2019-01-26 10:22:08               1
 9 VTS         2019-01-09 15:47:31 2019-01-09 15:54:03               1
10 VTS         2019-01-13 16:59:56 2019-01-13 17:14:49               1
# ℹ 7,657 more rows
# ℹ 18 more variables: trip_distance <dbl>, pickup_longitude <dbl>,
#   pickup_latitude <dbl>, rate_code <chr>, store_and_fwd <chr>,
#   dropoff_longitude <dbl>, dropoff_latitude <dbl>, payment_type <chr>,
#   fare_amount <dbl>, extra <dbl>, mta_tax <dbl>, tip_amount <dbl>,
#   tolls_amount <dbl>, total_amount <dbl>, improvement_surcharge <dbl>,
#   congestion_surcharge <dbl>, pickup_location_id <int>, …

How do you save a Parquet file?

Use write_parquet() from arrow (same name in nanoarrow!)

write_parquet(taxi, "new_taxi.parquet")

# fs <- S3FileSystem$create(anonymous = TRUE)
# df <- read_parquet(fs$path("arrow-datasets/nyc-taxi/year=2019/month=1/part-0.parquet"))
# write_parquet(df, "taxi.parquet")
taxi_big <- read_parquet("taxi.parquet")
nrow(taxi_big)

[1] 7667255

fs::file_size("taxi.parquet")

135M

library(tictoc)
tic()
readr::write_csv(taxi_big, "taxi_big.csv")
toc()

28.291 sec elapsed

fs::file_size("taxi_big.csv")

924M

tic()
from_csv <- readr::read_csv("taxi_big.csv")

Rows: 7667255 Columns: 22
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr   (4): vendor_name, rate_code, store_and_fwd, payment_type
dbl  (12): passenger_count, trip_distance, fare_amount, extra, mta_tax, tip_...
lgl   (4): pickup_longitude, pickup_latitude, dropoff_longitude, dropoff_lat...
dttm  (2): pickup_datetime, dropoff_datetime

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

toc()

6.914 sec elapsed

tic()
from_parquet <- read_parquet("taxi.parquet")
toc()

0.462 sec elapsed

tic()
just_times <- read_parquet("taxi.parquet", col_select = c(pickup_datetime, dropoff_datetime))
toc()

0.183 sec elapsed

zip_data <- data.frame(state = c("New York", "New Jersey"), zip = c("11213", "07001"))
zip_data

       state   zip
1   New York 11213
2 New Jersey 07001

write.csv(zip_data, "zip.csv", row.names = FALSE)
read.csv("zip.csv")

       state   zip
1   New York 11213
2 New Jersey  7001

"state","zip"
"New York","11213"
"New Jersey","07001"

library(readr)
read_csv("zip.csv")

Rows: 2 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): state, zip

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

# A tibble: 2 × 2
  state      zip  
  <chr>      <chr>
1 New York   11213
2 New Jersey 07001

write_parquet(zip_data, "zip.parquet")
read_parquet("zip.parquet")

# A tibble: 2 × 2
  state      zip  
  <chr>      <chr>
1 New York   11213
2 New Jersey 07001

library(ggplot2)
ggplot(diamonds, aes(x = cut)) + geom_bar() + theme_minimal()

library(readr)

write_csv(diamonds, "diamonds.csv")
diamonds_from_csv <- read_csv("diamonds.csv")

Rows: 53940 Columns: 10
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): cut, color, clarity
dbl (7): carat, depth, table, price, x, y, z

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

ggplot(diamonds_from_csv, aes(x = cut)) + geom_bar() + theme_minimal()

library(arrow)

write_parquet(diamonds, "diamonds.parquet")
diamonds_from_parquet <- read_parquet("diamonds.parquet")

ggplot(diamonds_from_parquet, aes(x = cut)) + geom_bar() + theme_minimal()

library(dplyr)

Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

starwars_subset <- starwars |> select(name, films)
starwars_subset

# A tibble: 87 × 2
   name               films    
   <chr>              <list>   
 1 Luke Skywalker     <chr [5]>
 2 C-3PO              <chr [6]>
 3 R2-D2              <chr [7]>
 4 Darth Vader        <chr [4]>
 5 Leia Organa        <chr [5]>
 6 Owen Lars          <chr [3]>
 7 Beru Whitesun Lars <chr [3]>
 8 R5-D4              <chr [1]>
 9 Biggs Darklighter  <chr [1]>
10 Obi-Wan Kenobi     <chr [6]>
# ℹ 77 more rows

starwars_subset |>
  write.csv("starsub.csv")

Error in `utils::write.table()`:
! unimplemented type 'list' in 'EncodeElement'

read.csv("starsub.csv")

Warning in read.table(file = file, header = header, sep = sep, quote = quote, :
incomplete final line found by readTableHeader on 'starsub.csv'

  X           name films
1 1 Luke Skywalker    NA

starwars_subset |>
  write_csv("starsub.csv")

starsub_csv <- read_csv("starsub.csv")

Rows: 87 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): name
lgl (1): films

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

starsub_csv$films[1]

[1] NA

starwars_subset |>
  write_parquet("starsub.parquet")

starsub_parquet <- read_parquet("starsub.parquet")

starsub_parquet$films[1]

<list<character>[1]>
[[1]]
[1] "A New Hope"              "The Empire Strikes Back"
[3] "Return of the Jedi"      "Revenge of the Sith"    
[5] "The Force Awakens"      

Reading Parquet metadata

You can do this using the nanoparquet package!

nanoparquet::read_parquet_info("taxi-mini.parquet")

# A data frame: 1 × 7
  file_name         num_cols num_rows num_row_groups file_size parquet_version
  <chr>                <int>    <dbl>          <int>     <dbl>           <int>
1 taxi-mini.parquet       22     7667              1    204709               1
# ℹ 1 more variable: created_by <chr>

nanoparquet::read_parquet_schema("taxi-mini.parquet")

# A data frame: 23 × 14
   file_name r_col name  r_type type  type_length repetition_type converted_type
   <chr>     <int> <chr> <chr>  <chr>       <int> <chr>           <chr>         
 1 taxi-min…    NA sche… <NA>   <NA>           NA REQUIRED        <NA>          
 2 taxi-min…     1 vend… chara… BYTE…          NA OPTIONAL        UTF8          
 3 taxi-min…     2 pick… POSIX… INT64          NA OPTIONAL        TIMESTAMP_MIL…
 4 taxi-min…     3 drop… POSIX… INT64          NA OPTIONAL        TIMESTAMP_MIL…
 5 taxi-min…     4 pass… double INT64          NA OPTIONAL        <NA>          
 6 taxi-min…     5 trip… double DOUB…          NA OPTIONAL        <NA>          
 7 taxi-min…     6 pick… double DOUB…          NA OPTIONAL        <NA>          
 8 taxi-min…     7 pick… double DOUB…          NA OPTIONAL        <NA>          
 9 taxi-min…     8 rate… chara… BYTE…          NA OPTIONAL        UTF8          
10 taxi-min…     9 stor… chara… BYTE…          NA OPTIONAL        UTF8          
# ℹ 13 more rows
# ℹ 6 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
#   precision <int>, field_id <int>, children <list>

nanoparquet:::read_parquet_pages("taxi-mini.parquet")

# A data frame: 44 × 14
   file_name         row_group column page_type       page_header_offset
   <chr>                 <int>  <int> <chr>                        <dbl>
 1 taxi-mini.parquet         0      0 DICTIONARY_PAGE                  4
 2 taxi-mini.parquet         0      0 DATA_PAGE                       34
 3 taxi-mini.parquet         0      1 DICTIONARY_PAGE               1423
 4 taxi-mini.parquet         0      1 DATA_PAGE                    48607
 5 taxi-mini.parquet         0      2 DICTIONARY_PAGE              61281
 6 taxi-mini.parquet         0      2 DATA_PAGE                   108551
 7 taxi-mini.parquet         0      3 DICTIONARY_PAGE             121227
 8 taxi-mini.parquet         0      3 DATA_PAGE                   121278
 9 taxi-mini.parquet         0      4 DICTIONARY_PAGE             124298
10 taxi-mini.parquet         0      4 DATA_PAGE                   128776
# ℹ 34 more rows
# ℹ 9 more variables: uncompressed_page_size <int>, compressed_page_size <int>,
#   crc <int>, num_values <int>, encoding <chr>,
#   definition_level_encoding <chr>, repetition_level_encoding <chr>,
#   data_offset <dbl>, page_header_length <int>

Disclaimer: prose below here was generated with Claude, but checked by me.

Arrow vs. Parquet vs. Feather

This is the distinction that confuses everyone:

• Parquet = a file format for storing data on disk (columnar, compressed, self-describing)
• Arrow columnar format = a specification for how columnar data is laid out in memory - designed for zero-copy reads and fast analytics. Multiple implementations exist across languages (C++, Rust, Java, Go, etc.)
• Arrow IPC format = a way to write Arrow-formatted data to disk or send it between processes, preserving the in-memory layout. Very fast to read because there’s no decoding step - the data on disk is already in Arrow format
• Feather = an older name for Arrow IPC files (V1 is deprecated; V2 = Arrow IPC)
• The {arrow} R package = an R interface to the Arrow C++ library, giving you tools to read/write Parquet, Arrow IPC, and CSV files, plus a dplyr backend for larger-than-memory data

In practice: Parquet is the best default for storing and sharing data. Arrow IPC can be faster to read/write but files are larger and less widely supported outside the Arrow ecosystem.

What you can and can’t store in Parquet

All standard R types (numeric, integer, character, logical, Date, POSIXct), list columns, and factor levels can be stored in Parquet. Custom R classes are preserved via metadata when round-tripping in R.

Nested data example - list columns survive the round-trip:

nested_data <- tibble(
  species = c("cat", "dog", "bird"),
  sounds  = list(
    c("meow", "purr", "hiss"),
    c("woof", "bark", "growl"),
    c("tweet", "squawk")
  )
)

write_parquet(nested_data, "nested.parquet")
read_parquet("nested.parquet")

# A tibble: 3 × 2
  species            sounds
  <chr>   <list<character>>
1 cat                   [3]
2 dog                   [3]
3 bird                  [2]

What you can’t store:

• Arbitrary R objects (ggplot objects, model fits, environments)
• Everything must map to Parquet’s type system
• Custom R class metadata is R-specific (won’t auto-translate to Python)

Partitioning

Partitioning splits a dataset into separate files based on column values. When you filter, Arrow only reads the relevant files.

write_dataset(
  taxi_big,
  path = "taxi-partitioned",
  format = "parquet",
  partitioning = "payment_type"
)

list.files("taxi-partitioned", recursive = TRUE)

[1] "payment_type=Cash/part-0.parquet"         
[2] "payment_type=Credit%20card/part-0.parquet"
[3] "payment_type=Dispute/part-0.parquet"      
[4] "payment_type=No%20charge/part-0.parquet"  

taxi_ds <- open_dataset("taxi-partitioned")

taxi_ds |>
  filter(payment_type == "Credit card") |>
  summarise(avg_tip = mean(tip_amount, na.rm = TRUE)) |>
  collect()

# A tibble: 1 × 1
  avg_tip
    <dbl>
1    2.55

Rules of thumb for partitioning:

• Partition on columns you frequently filter by
• Aim for individual files between 20 MB and 2 GB
• Avoid more than ~10,000 partition files

Bonus: analysing with Arrow

You can use dplyr verbs on an Arrow dataset - the work happens in Arrow, not R, until you collect().

taxi_ds <- open_dataset("taxi-partitioned")

taxi_ds |>
  filter(trip_distance > 0, fare_amount > 0) |>
  group_by(payment_type) |>
  summarise(
    n = n(),
    avg_fare = mean(fare_amount, na.rm = TRUE),
    avg_tip = mean(tip_amount, na.rm = TRUE)
  ) |>
  collect()

# A tibble: 4 × 4
  payment_type       n avg_fare  avg_tip
  <chr>          <int>    <dbl>    <dbl>
1 Credit card  5461817     12.5 2.54    
2 Cash         2112585     11.3 0.000315
3 Dispute         7498     13.9 0.00550 
4 No charge      24026     39.3 0.00254 

For much more on this, see Scaling Up With R and Arrow.

Where can I read more?

• Scaling Up With R and Arrow - free online book
• R for Data Science: Arrow chapter
• Arrow R package docs
• posit::conf(2024) Arrow workshop
• nanoparquet

Background

The Arrow R package contains bindings to dplyr functions, which means that users of arrow can work on their data using the same functions they’re used to using in dplyr.

In dplyr version 1.2.0, released in February 2026, some new functions were added; one example is recode_values(). You can use it to create a new column by matching values and mapping them to replacements; for example, turning numeric codes into labels. Here’s an example:

library(dplyr)

likert <- tibble(score = c(1, 3, 5, 2, 4))

likert |>
  mutate(
    category = score |>
      recode_values(
        1 ~ "Strongly disagree",
        2 ~ "Disagree",
        3 ~ "Neutral",
        4 ~ "Agree",
        5 ~ "Strongly agree"
      )
  )

# A tibble: 5 × 2
  score category         
  <dbl> <chr>            
1     1 Strongly disagree
2     3 Neutral          
3     5 Strongly agree   
4     2 Disagree         
5     4 Agree            

I wanted to create bindings to these in Arrow, but they can be fiddly though. In the past, to complete this task, I’ve typically ended up reading the code in the dplyr package, and then stepping through it one line at a time to understand it in depth, and then writing the arrow equivalent. It’s deep work and can be quite time-consuming, depending on how complex it is.

I wanted to see if I could use Claude Code to simplify this process.

What I did

Result

You can see the final PR here.

This took lots of rounds of manual review myself, but ultimately it meant that there was minimal review from others when I marked it “ready for review”. While I think it’s fine to use AI tools for code changes and PRs like this, this shouldn’t end up adding extra burden for the reviewer, and I wanted to make sure I really had properly read and engaged with the code before passing it onto someone else.

I’m not naturally good at this and have made tons of “obvious mistakes” in the past on entirely human-generated PRs - sometimes the feels of “being done” with a hard or long-running task is just too appealing. Wanting to avoid “AI laziness”, I made a real effort to tackle this here, and tried techniques like leaving lots of time between working intensely on the code and reviewing it myself. Resetting my mental energy really helped.

I still found it hard to read every line, but having done so many iterative manual reviews and focusing on testing early gave me confidence that I’d done a proper job of this. Honestly though, the fact I’d worked on this bit of the codebase without AI tools previously really helped.

Future plans

I have a really exciting new project coming up soon, where I’ll be working on a new codebase, and I don’t plan to use AI for all of it. I’m planning on reading code and writing notes myself, and trying first to fix a few bugs manually to understand the paths through the code, using tools like Claude to help me understand the source rather than to generate it, until I understand it better. I’ll write another blog post on the topic in a few months.

Background

The Apache Arrow repository contains multiple implementations of Arrow. Both Python and R implementations are wrappers around the C++ implementation. Although many developers work across multiple languages, triage of new issues tends to be done by individuals more aligned to an individual language.

We use labels in the Apache arrow repository to identify which implementation an issue relates to. Our issue creation workflow does make users select a component (e.g. “Python”, “R”, “C++” etc), but on occasion, new issues are opened by users via a different route with the component label missing.

The problem here is that many maintainers will filter issues by component and so unlabeled issues can end up completely ignored. The problem that I want to solve here is whether we can automatically classify issues by component by passing the text in the issue to an LLM, so that we can potentially automate labelling of new issues so they get responded to appropriately.

Loading the data

The first thing that I need to do is get hold of the data. I wrote a script that retrieves issues from the GitHub API and caches them in Parquet files. Parquet is an excellent format for this because it is efficient for storage in terms of space, and unlike CSV files allows nested columns. This means that when you retrieve data from an API in JSON format there’s potentially less work to do to get it into a good state for storage.

Here’s a quick preview of the dataset.

The important column here for us is the labels column, which is a list column containing 0 or more labels. R, Python, and C++ issues all have labels like “Component: Python”. Although the issue titles have the component at the start of their title and also included in their body, this is automatically added based on the label, so isn’t helpful to our problem.

My plan is to extract a subset of C++/Python/R component tickets, remove the autogenerated “component” label that gets added to the body of the ticket, and see if I can get an LLM to classify the component based on the content of the issue body.

Let’s extract a subset of ~30 of each.

library(dplyr)
target_labels <- c("Component: C++", "Component: R", "Component: Python")

issues_with_components <- data |>
    select(number, title, body, labels) |>
    # make sure we match our target label
    filter(purrr::map_lgl(labels, ~ any(.x %in% target_labels))) |>
    # make sure we have code in the PR body
    filter(stringr::str_detect(body, "```")) |>
    # create new column containing only component labels
    mutate(component = purrr::map(labels, ~ .x[.x %in% target_labels])) |>
    # filter to only keep rows where we have a single component
    filter(purrr::map_int(component, length) == 1) |>
    # convert from list to character column
    mutate(component = purrr::map_chr(component, 1)) |>
    # Remove "Component prefix"
    mutate(component = stringr::str_remove(component, "Component: ")) |>
    # Remove the bit in the body where the "component" has been auto-appended
    mutate(body = stringr::str_remove(body, "### Component\\(s\\).*$"))

I want to create a smaller data set to test the LLM classification on simply because I don’t have any idea at this point on how long it’ll take or how much it’ll cost. The next step is to create a dataset containing 30 issues from each component.

I’ll store this intermediate version to work with later too.

r_issues <- issues_with_components |>
    filter(component == "R") 

python_issues <- issues_with_components |>
    filter(component == "Python") 

cpp_issues <- issues_with_components |>
    filter(component == "C++") 

issues_dataset <- bind_rows(
    r_issues |> slice_head(n = 30),
    python_issues |> slice_head(n = 30),
    cpp_issues |> slice_head(n = 30)
)

# shuffle the dataset
issues_dataset <- issues_dataset |>
     slice_sample(prop = 1)

# A tibble: 90 × 5
   number title                                           body  labels component
    <dbl> <chr>                                           <chr> <chr>  <chr>    
 1  49129 "[R] Compilation fails when Parquet support is… "###… <chr>  R        
 2  47801 "[Python][CI] Remove use of CMAKE_POLICY_VERSI… "###… <chr>  Python   
 3  49222 "[Python] data loss converting to Table from p… "###… <chr>  Python   
 4  48265 "[Python][GPU] Numba interop tests broken by N… "###… <chr>  Python   
 5  43336 "[R] stringr binding for `str_starts` fails ba… "###… <chr>  R        
 6  48349 "[Python] Support array indices in pc.list_ele… "###… <chr>  Python   
 7  44306 "[R] please write unregister_scalar_function a… "###… <chr>  R        
 8  47698 "pc.subtract_checked(x, x) with x as pa.array(… "###… <chr>  Python   
 9  48832 "[R] arrow::write_parquet error with zero-leng… "###… <chr>  R        
10  45314 "[R] arrow R package: multiple replacement dis… "###… <chr>  R        
# ℹ 80 more rows

LLM classification

Before we properly get started, I’m going to have a look at an individual example.

Let’s have a quick look at the first issue in the data set

issues_dataset |> slice(1)

# A tibble: 1 × 5
  number title                                            body  labels component
   <dbl> <chr>                                            <chr> <chr>  <chr>    
1  49129 [R] Compilation fails when Parquet support is d… "###… <chr>  R        

Ok, so I can see from the title that it’s an R function. I’ll paste it below so you can see it.

### Describe the bug, including details regarding any error messages, version, and platform.

When building the R bindings with Parquet support disabled (`ARROW_R_WITH_PARQUET` not defined), compilation fails with errors about undeclared 'parquet' identifiers in arrowExports.cpp.

The issue occurs because Parquet-related code in dataset.cpp and arrowExports.cpp lacks proper preprocessor guards. Specifically:

1. In arrowExports.cpp (line ~1871), the function declaration for `dataset___ParquetFileWriteOptions__update` uses `parquet::WriterProperties` and `parquet::ArrowWriterProperties` types without checking if `ARROW_R_WITH_PARQUET` is defined.

2. In dataset.cpp, Parquet-related includes and function implementations are not guarded by `ARROW_R_WITH_PARQUET` checks.

Compilation error:

```
arrowExports.cpp:1871:131: error: use of undeclared identifier 'parquet'
```

Version: 23.0.0
Platform: Linux

### Component(s)

R

And now let’s try classifying it. Here I’m going to start a new conversation, and then use structured output so that I can determine the exact response type I get from the LLM. In this case I’m using enumerated values so I can guarantee that we get one of the pre-specified values back and nothing else.

library(ellmer)
chat <- chat_anthropic()

Using model = "claude-sonnet-4-5-20250929".

type_language <- type_enum(values = c("Python", "C++", "R"), "The language implementation of Arrow that the issue relates to")

chat$chat_structured(issues_dataset$body[1], type = type_language)

[1] "R"

Great! This was successful so now I’m going to try running it on the rest of the data set. In the above code I just use the default model that ellmer is configured to use, which turned out to be Claude Sonnet 4.5, but actually I want to use the cheapest option to see how well it performs.

Let’s have a look at which Anthropic models are available to us in ellmer.

models_anthropic()

                          id              name created_at cached_input input
1          claude-sonnet-4-6 Claude Sonnet 4.6 2026-02-17           NA    NA
2            claude-opus-4-6   Claude Opus 4.6 2026-02-04           NA    NA
3   claude-opus-4-5-20251101   Claude Opus 4.5 2025-11-24           NA    NA
4  claude-haiku-4-5-20251001  Claude Haiku 4.5 2025-10-15         0.10  1.00
5 claude-sonnet-4-5-20250929 Claude Sonnet 4.5 2025-09-29         0.30  3.00
6   claude-opus-4-1-20250805   Claude Opus 4.1 2025-08-05         1.50 15.00
7     claude-opus-4-20250514     Claude Opus 4 2025-05-22         1.50 15.00
8   claude-sonnet-4-20250514   Claude Sonnet 4 2025-05-22         0.30  3.00
9    claude-3-haiku-20240307    Claude Haiku 3 2024-03-07         0.03  0.25
  output
1     NA
2     NA
3     NA
4   5.00
5  15.00
6  75.00
7  75.00
8  15.00
9   1.25

I’m gonna go for the Haiku model and this time I’m gonna do things a little bit differently to do the classification across the whole data set.

I could try writing a loop of some sort, but it’ll be slow, and so I’m gonna use the function parallel_chat_structured() so that I can start a new conversation for each individual issue classification task, and run them in parallel so it takes less time.

haiku_chat <- chat_anthropic(model = "claude-3-haiku-20240307")

haiku_classified <- parallel_chat_structured(
    chat = haiku_chat,
    prompts = as.list(issues_dataset$body),
    type = type_language
)

I was surprised to see a message “waiting 35s for rate limiting” when it ran but after diving deeper into the Claude developer docs I realised that I’d hit the 50 requests per minute limit. Pretty simple to limit in the function call, but not need as waiting wasn’t a problem.

I also wanted to check how much it had cost - $0.04 in total, and some of this was earlier experimentation I had done in the process.

token_usage()

   provider                      model    input output cached_input price
1 Anthropic claude-sonnet-4-5-20250929   3795.0    503            0 $0.02
2 Anthropic    claude-3-haiku-20240307 136299.5   2996            0 $0.04

Now I’d run it successfully with Haiku, I wanted to check the quality of the results. I’m just going to use a very rough measure of how many were a perfect match.

issues_dataset$llm_component <- haiku_classified
mean(issues_dataset$llm_component == issues_dataset$component, na.rm = TRUE) |> round(2)

[1] 0.97

OK, so this is great; 97% accuracy!

And how about the ones it got wrong? Let’s take a look!

issues_dataset |>
    filter(llm_component != component)

# A tibble: 3 × 6
  number title                              body  labels component llm_component
   <dbl> <chr>                              <chr> <chr>  <chr>     <fct>        
1  47902 [C++][Acero] Add support for `Str… "###… <chr>  C++       Python       
2  48761 [C++] Substrait serialised expres… "###… <chr>  C++       Python       
3  49214 [C++][Compute] Allow reusing hash… "###… <chr>  C++       Python       

OK, so 3 C++ tickets incorrectly classified as Python. Let’s have a look at each issue body.

### Describe the enhancement requested

Similarly to #43716, it would be useful to support fields with struct types in joins.

Here is similar pyarrow code to that shared by Anja in the other issue to illustrate this case:
```
import pyarrow as pa
import pyarrow.acero as acero

# ---

table_1 = pa.table({"a": [1, 2, 3], "b": ["x", "y", "z"]})
table_1_node = acero.Declaration(
    "table_source", options=acero.TableSourceNodeOptions(table_1)
)

table_2 = pa.table(
    {"a": [1, 2, 3], "c": [{"x": 1, "y": 2}, {"x": 2, "y": 2}, {"x": 3, "y": 2}]}
)
table_2_node = acero.Declaration(
    "table_source", options=acero.TableSourceNodeOptions(table_2)
)

expected = pa.table(
    {
        "a": [1, 2, 3],
        "b": ["x", "y", "z"],
        "c": [{"x": 1, "y": 2}, {"x": 2, "y": 2}, {"x": 3, "y": 2}],
    }
)

# ---

hash_join_options = acero.HashJoinNodeOptions(
    "left outer", left_keys=["a"], right_keys=["a"]
)

join_node = acero.Declaration(
    "hashjoin", options=hash_join_options, inputs=[table_1_node, table_2_node]
)

result = join_node.to_table()

assert result == expected
```
When run, the join operation raises the following error:
```
    result = join_node.to_table()
  File "pyarrow/_acero.pyx", line 592, in pyarrow._acero.Declaration.to_table
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Data type struct<x: int64, y: int64> is not supported in join non-key field c
```

I _think_ addressing this may depend on #45001, but I am not positive

### Component(s)

C++

Ok, so here the issue is the fact that the problem was detected in the Python library, which wraps the C++ library. The user who opened this ticket had a sophisticated enough level of knowledge to understand that link, and so rather than classifying this as an incorrect response, I’d actually probably call this example not typical of where we actually want to apply the model and so just not a good example for our test data set. Most unlabelled tickets come from new users who don’t have a deep understanding of the interplay between the different components. Let’s take a look at another example.

### Describe the bug, including details regarding any error messages, version, and platform.

When serialising pyarrow expressions using substrait, the size of the serialised buffer increases exponentially with the number of binary operation clauses in the expression due to extension URIs being repeated again for each nested expression.  

Testing with pyarrow 21.0.0: 
```
================================================================================
Testing with 1 OR condition(s)
================================================================================
Expression:  (int_col == 1)
Substrait size: 201 bytes
Total extension URIs: 1

================================================================================
Testing with 2 OR condition(s)
================================================================================
Expression:  ((int_col == 1) or (int_col == 2))
Substrait size: 634 bytes
Total extension URIs: 5

================================================================================
Testing with 4 OR condition(s)
================================================================================
Expression:  ((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4))
Substrait size: 2967 bytes
Total extension URIs: 29

================================================================================
Testing with 6 OR condition(s)
================================================================================
Expression:  ((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6))
Substrait size: 12017 bytes
Total extension URIs: 125

================================================================================
Testing with 8 OR condition(s)
================================================================================
Expression:  ((((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6)) or (int_col == 7)) or (int_col == 8))
Substrait size: 48306 bytes
Total extension URIs: 509

================================================================================
Testing with 10 OR condition(s)
================================================================================
Expression:  ((((((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6)) or (int_col == 7)) or (int_col == 8)) or (int_col == 9)) or (int_col == 10))
Substrait size: 193172 bytes
Total extension URIs: 2045

================================================================================
Testing with 12 OR condition(s)
================================================================================
Expression:  ((((((((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6)) or (int_col == 7)) or (int_col == 8)) or (int_col == 9)) or (int_col == 10)) or (int_col == 11)) or (int_col == 12))
Substrait size: 772342 bytes
Total extension URIs: 8189

================================================================================
Testing with 14 OR condition(s)
================================================================================
Expression:  ((((((((((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6)) or (int_col == 7)) or (int_col == 8)) or (int_col == 9)) or (int_col == 10)) or (int_col == 11)) or (int_col == 12)) or (int_col == 13)) or (int_col == 14))
Substrait size: 3105111 bytes
Total extension URIs: 32765

================================================================================
Summary:
================================================================================
 OR Conditions  |  Size (bytes)   |    Size (KB)    | Extension URIs 
----------------------------------------------------------------------
       1        |       201       |      0.20       |        1       
       2        |       634       |      0.62       |        5       
       4        |      2,967      |      2.90       |       29       
       6        |     12,017      |      11.74      |       125      
       8        |     48,306      |      47.17      |       509      
      10        |     193,172     |     188.64      |      2045      
      12        |     772,342     |     754.24      |      8189      
      14        |    3,105,111    |     3032.33     |      32765     
```
 
Serialization was done by simply: `buf = pyarrow.substrait.serialize_expressions([expr], ["result"], schema)`

Unfortunately, it might not be possible to just use `is_in` in the expression for my use-case so that might not be a possible workaround. 

Would be nice to know if I am somehow misusing the API which is causing this. Thanks! 

cc: @westonpace 

### Component(s)

C++

Again, a more complex example, identified in PyArrow but relating to C++.

And the third?

### Describe the enhancement requested

### Problem

The `is_in` kernel (and other set lookup operations) rebuilds the internal hash table (MemoTable) from `value_set` on **every invocation**, even when `value_set` is identical across calls. There is currently no API to pre-build and reuse the hash table.

This becomes a significant bottleneck when the same `value_set` is used to filter many arrays (e.g., in a streaming/batch processing pipeline).

### Use Case

In Ray Data Checkpointing, we want to use `pc.is_in` to perform an anti-join: filtering out already-processed row IDs from each incoming block of data. The checkpoint ID set (`value_set`) is the same across all blocks, but the hash table is rebuilt for every block.

```python
# value_set is the same across all calls — hash table is rebuilt each time
for block in blocks:
    membership = pc.is_in(block["id"], value_set=checkpoint_ids)
    filtered = block.filter(pc.invert(membership))
```

With 100M checkpoint IDs and 20 blocks, the hash table construction alone takes **~250s** (extrapolating from #36059 benchmarks), doing identical work 20 times.

### Benchmark

Building on the benchmarks from #36059, the cost of rebuilding becomes clear when multiplied across calls:

```python
import pyarrow as pa
import pyarrow.compute as pc
import numpy as np

value_set = pa.array(np.arange(10_000_000))
blocks = [pa.array(np.random.choice(np.arange(1000), 100)) for _ in range(20)]

# Current: ~25s total (rebuilds hash table 20 times)
for block in blocks:
    pc.is_in(block, value_set)

# Ideal: ~1.3s (build once, probe 20 times)
```

### Proposed API

**Option A — Prepared/materialized value set object:**

```python
# Build hash table once
lookup = pc.SetLookup(value_set)  # or pc.prepare_set(value_set)

# Reuse across calls — probes only, no rebuild
for block in blocks:
    mask = lookup.is_in(block)
```

**Option B — Caching in SetLookupOptions:**

```python
options = pc.SetLookupOptions(value_set, cache=True)

# First call builds hash table; subsequent calls reuse it
for block in blocks:
    mask = pc.is_in(block, options=options)
```

**Option C — Lower-level hash set exposure:**

```python
hash_set = pc.HashSet(value_set)  # builds MemoTable

for block in blocks:
    mask = hash_set.is_in(block)
    # or: mask = pc.is_in(block, options=pc.SetLookupOptions(hash_set))
```

### Impact

This would benefit any workload that probes the same set repeatedly:

- Streaming anti-joins / semi-joins
- Incremental processing (checkpoint filtering)
- Repeated dictionary lookups
- Any ETL pipeline filtering against a static blocklist/allowlist

### Related

- #36059 — Performance of building up HashTable (MemoTable) in `is_in` kernel (addressed MemoTable speed, but not reuse)


### Component(s)

C++

Same again!

Given that these kinds of issues are typically reported by more experienced contributors and don’t end up going unlabelled anyway, we could justifiably deem them to be poor examples for the task at hand and so claim 100% accuracy here.

I’m calling this experiment a success and my next steps are to see if I can deploy this or something similar to help with other aspects of Apache Arrow maintenance. The main thing I took from this is how accessible LLM-powered tooling has become and how simple it has been to use for an otherwise manual task.

I’ve just finished writing a course about how to work with ellmer and extract tidy data from text, and create chatbots; if this is something you’re interested in learning about with step-by-step tutorials, you can find my course at: https://www.aifordatapeople.com/courses/llms-in-r.

Attitudes to AI tools

Earlier reactions to the use of AI tools in some open source projects have been quite negative, and there are lots of examples of open source projects explicitly banning the use of AI tools in generating contributions, for example, Gentoo Linux in 2024 banned all AI contributions.

I have some sympathy for this perspective, and there are certainly good and valid reasons around complications relating to licensing and AI-generated code that I believe open source developers need to be cautious of. The ASF have published some high-level guidelines on the use of generative tooling that capture the risk of replicating copyrighted materials.

That said though, I also believe that the evolution of these tools means that condemning them outright is throwing the metaphorical baby out with the bathwater. Entire features created with AI absolutely can be problematic from a licensing perspective, but much of the maintenance work required for existing codebases is less fraught. In my experience, fixes and changes generated based on the existing codebase and reasoning around it rarely end up replicating other codebases, and these tools are a net win for maintainers when used well.

As the sophistication of tools has increased, I found that at some point in 2025, the balance tipped for me from relatively poor results towards them becoming net useful. I also believe a lot of developers are in this place where many of us are actively using these tools but fewer are openly admitting it, due to the scepticism out there.

While there is some validity in the more cautious perspectives, it’s unrealistic to pretend that they aren’t being used pretty widely at this point.

The problem of AI-generated PRs in open source

There is a certain class of AI-generated pull request that is becoming problematic. There have a distinct pattern to them:

• a pull request is submitted by a contributor who has never previously contributed to the codebase
• code changes are often accompanied by verbose and unnecessary comments
• the scope of the changes is overly broad scope
• unit tests are either excessive or non-existent
• the code itself doesn’t match the style or conventions of the other code
• the contributor doesn’t ask any questions about anything they’re unsure about

When I see a pull request with these characteristics, I tend to assume that the contributor wanted to contribute to the codebase for whatever reason, decided to use AI to generate the code, but then didn’t engage much if at all with the output before submitting the PR.

To a certain extent I get it. Some of my first open source pull requests were absolutely done for the sake of something that I viewed as a real achievement and not a genuine investment in a codebase, because I didn’t really understand how open source worked at that point. I was fortunate to end up interacting with patient people who took their time to guide me through the process and help me learn, for which I’m grateful.

And I’ve made a fair share of embarrassing mistakes in my own use of AI tools. In my earlier explorations, I accidentally allowed Claude to push to an upstream branch instead of my own fork, and proposed poorly thought-through changes that I didn’t actually understand well enough to take responsibility for when other maintainers responded with thoughtful questions. The realisation of this potential for harm made me drastically alter my approach, but I had to make the mistake to learn the lesson.

All of that said though, most open source maintainers are pressed for time to work on the project. It’s great working with new contributors, both for the human connection and because it improves project health by increasing the number of people who are able to contribute, but there is very little value in a maintainer reviewing AI-generated code which the author has either abandoned or will just continue to use AI to respond to the provided suggestions.

Thoughtful use of AI tools requires multiple iterations and changes, and questioning of the tooling as to the reasoning behind changes, and taking ownership of the changes through the understanding gained during the process.

Taking it to the mailing list

After noticing the increase in AI generated PRs, I wanted to validate the preferred approach to handling these PRs with the wider Arrow community, and so I started a discussion on the Arrow developer mailing list about defining a set of community-agreed guidelines for AI-generated PRs.

It takes a chunk of emotional energy trying to respond to a PR with the empathy and welcoming nature that we want to show towards new contributors, while simultaneously not investing too much time and energy in a pull request which may well be abandoned. Conversely though, if the contributor really is interested in engaging, we don’t want to drive them away.

In reality, we can’t necessarily tell when people make thoughtful use of AI tooling, and so my own personal motivation for suggesting this guidance was to have an official source that we can link to during reviews of PRs which appear to be AI generated.

Given all the negativity I’ve seen online around the use of AI tooling, and also my own use of such tools, I was a little nervous to see what the conversation would look like when I opened the discussion on the Apache Arrow developer mailing list.

It was great to see that, as ever, the Apache Arrow developer community engaged in a thoughtful discussion in which folks suggested additions to my suggested phrasing of policy, and chimed in with their own experiences. This clearly isn’t a novel or new idea, and Andrew Lamb pointed me straight to some excellent guidelines which already existed in the DataFusion project. (For context, DataFusion is a query engine built on Arrow, used to be kept in the former Arrow monorepo, and is part of the Arrow community). I borrowed some ideas from the DataFusion guidelines as well as the contributions from other folks on the mailing list. The key recommendations we ended up with were:

• Only submit a PR if you are able to debug and own the changes yourself - review all generated code to understand every detail
• Match the style and conventions used in the rest of the codebase, including PR titles and descriptions
• Be upfront about AI usage and summarise what was AI-generated
• If there are parts you don’t fully understand, leave comments on your own PR explaining what steps you took to verify correctness
• Watch for AI’s tendency to generate overly verbose comments, unnecessary test cases, and incorrect fixes
• Break down large PRs into smaller ones to make review easier

Since we had this discussion, the PR that we merged was linked to by a couple of other projects and after following the links back I found even more open source projects which had already had engaged in similar discussions or recent policy updates, for example, Jupyter, Zulip, LLVM, and Python.

Other challenges

Pull requests aren’t the only challenge that open source is seeing in terms of AI tool usage; multiple projects have also recently experienced an uptick in security reports. Some of these are valid, but others appear to constitute things that would be considered bugs or edge cases. While deploying fixes for these may genuinely improve the codebase, some examples appear to include strategic phrasing to depict bugs as security incidents, and feel a little disingenuous.

This is becoming more prevalent, as shown by the curl project recently closing its bug bounty program, and similar issues experienced by Apache Log4j.

The irony of the spurious reporting of alleged potential Denial-of-Service attacks is delightfully captured by curl author Daniel Stenberg in the abstract of his upcoming FOSDEM talk:

“Sloppy humans causing Denial-of-Service attacks by overloading maintainers with quickly produced almost-real-looking rubbish”.

AI is having a significant effect on open source development, some of it positive, but it’s certainly not without real challenges as well.

Context - adding a feature to daisyuiwidget

I’m currently collaborating with Charlotte Hadley on an R package called daisyuiwidget, which is a htmlwidgets package.

library(DT)

packages <- data.frame(
    package = c("htmlwidgets", "leaflet", "plotly", "DT"),
    first_release = as.Date(c("2014-12-09", "2015-06-24", "2015-11-17", "2015-06-09")
    )
)

DT::datatable(packages)

Aside - the stretch zone

I’m going to be talking at Posit Conf about increasing productivity with LLMs and a central thesis of my talk is the idea that LLMs are most useful when working on tasks just outside of your comfort zone, sometimes described as your stretch zone.

I’m defining the “stretch zone” in this context as something that isn’t within your current knowledge, but doesn’t seem so challenging that you have doubts about whether you can find a solution.

This task seemed like a great fit for this in terms of my past experience:

✅ Extensive Shiny app development experience

✅ Experience using row selection in DT-based apps to update other components

✅ Some JavaScript experience

❌ Very limited experience of how to implement htmlwidgets package from scratch

❌ Never implemented Shiny reactivity for UI components before

Learning with LLMs

I’d been reading a paper by Dr Xi Lin about the idea of using LLMs in self-directed learning.

Lin, X. (2023). Exploring the Role of ChatGPT as a Facilitator for Motivating Self-Directed Learning Among Adult Learners. Adult Learning, https://doi.org/10.1177/10451595231184928

Dr Lin discusses a self-directed learning framework proposed by Kim et al (2014) with the following aspects:

1. Establishing Learning Goals
2. Locate and Access Resources
3. Adopt and Execute Learning Activities
4. Monitor and Evaluate Performance
5. Reassess Learning Strategies

This framework can be used with ChatGPT to help support adult learners, and Dr Lin goes on to walk through a hypothetical example where an adult learner is using ChatGPT in each of those steps and discusses the kind of help that could be given.

I wanted to try some of this out, and so I created LLM summaries of each of the steps including Dr Lin’s suggestions of how an LLM might implement them, and used these summaries as prompts in a learning session with ChatGPT. I focused solely on steps 1 to 3 as the learning goal was an individual task rather than an ongoing programme.

Step 1 - Establish Learning Goals

Duration: 7 minutes

I wrote a little context about what I was trying to achieve, and pasted that and my summary of step 1, into ChatGPT.

ChatGPT asked me five questions:

• what my goals were
• my past experience
• exactly what it was that I was trying to achieve
• my preferred end state
• how long I wanted to spend working on this

It took my answers and generated a primary learning goal with supporting objectives.

Primary Learning Goal:

Within the next few hours, I want to understand and implement the necessary code to make my existing htmlwidget timeline respond to user interaction in a Shiny app—specifically, capturing a selected event and updating a variable on the server side.

Supporting Objectives:

• Identify how event selection can be captured in JavaScript and passed to Shiny via Shiny.setInputValue().
• Modify the widget’s JS code to emit a Shiny input when a user selects an event.
• Test and verify that the input value is received in Shiny and reflects the expected event ID or index.
• Write a minimal working Shiny app that uses the widget and displays the selected event value.

I feel that the value I got here from incorporating ChatGPT was clarity. The resulting learning goal was more precise than my original stated aim, and the supporting objectives made for concrete steps to aim for in pursuit of the overall goal.

Step 2 - Locate and Access Resources

Duration: 11 minutes

The second stage was identifying resources for learning.

It started smoothly - ChatGPT asked me if I preferred to learn by reading, watching, or doing, and when I replied “doing” it assured me that it would find code-first resources with working examples - great!

It gave me some options about kinds of examples I was interested in, and asked which htmlwidgets packages I was more familiar with so we could use those as examples to look at. At this point it suggested some initial links, but 2 out of the 3 didn’t actually work.

When I complained about this, it gave me some code examples to try out, which I pushed back on, stating I wanted resources to learn from and not ChatGPT-generated code,

At this point, it gave me a list of resources, divided into categories and a summary of what it had provided. The links supplied here did all work this time, but there was too much information, and a real risk of overwhelm.

I began to skim through a few articles, before I remembered that the aim of this step was only to identify resources, and not to deep dive into them quite yet, and so instead I moved onto the next step.

Step 3 - Adopt and Execute Learning Activties

Duration: 36 minutes

The third and final step was the most useful by far!

Once again, I pasted in the guide to this step, and ChatGPT created a plan for me containing four items:

1. Reverse-engineer one working widget
2. Build minimal standalone Shiny demo
3. Add minimal event tracking to your widget
4. Wire the widget into a basic Shiny app

This was actually pretty impressive; each step contained a goal, the steps required to achieve it, suggestions for how an LLM could help, and a timebox of how long to spend on each activity.

Here’s the first activity suggested:

🔍 1. Reverse-engineer one working widget

Goal: Understand how an existing widget (e.g. DT) captures selection and sends it to Shiny.

Activity:

• Open the DT GitHub repo.

• Locate where Shiny.setInputValue() is used in the JS files (inst/htmlwidgets/datatables.js).

• Make brief notes:

  • What JS event triggers the update?
  • How is the input ID constructed?
  • What data is passed to Shiny?

LLM Role:

• Ask ChatGPT to walk you through unfamiliar lines or patterns (e.g. “What’s this event handler doing?” or “What does this argument to setInputValue mean?”).

Timebox: 20–30 minutes.

I walked through each activity, setting a timer on my phone. I found that I completed each in less than the allotted time. I skipped the last activity, as I only required 1-3 to complete the task.

There was a lot of interaction between me and ChatGPT here, and I’m not going to paste it all, but essentially, during the time I used it to:

• explain some JavaScript code I didn’t understand by walking me through it line-by-line
• get clarity on how to implement some of its suggestions by asking exactly what it meant
• multiple rounds of debugging code I had written which didn’t work
• helping me work out exactly where the change was needed in my own code

In the last example, ChatGPT didn’t just tell me where to put the code, but walked through the important concepts to reason through the answer. It then asked me questions so that I could work out where in the code the change needed to go. Here’s a snippet from that interaction.

🤔 So… where do you want interactivity?

Let me ask you this:

Out of everything that’s being created here, what part of the DOM would the user actually click on if they were selecting a timeline event?

Once we know that, we can talk about how and where to attach the event handler.

Exactly—that’s the actionable node.

That means you’ll want to attach a JS event handler (e.g. .onclick or .addEventListener('click', ...)) to each .timeline-box as it’s created.

Now a follow-up:

When you’re looping over x.events, what info do you need to send back to Shiny when a .timeline-box is clicked? (Think: what identifier, value, or payload do you want to capture?)

We carried on the back and forth like this for a while, with ChatGPT asking me questions to guide me towards the solution. If I got an answer wrong, it replied with an encouraging tone and explained why I was incorrect, and the specific piece of knowledge I was missing.

Eventually, we go to the end of our interaction.

It sounds like a lot, but this whole experience took around 1 hour from end to end which I found pretty impressive.

The final output

Here is the final PR: https://github.com/thisisnic/daisyuiwidget/pull/11. It’s not particularly complicated, but in the hour it took to write, I learned about some of the internals of the DT package, created a standalone example of sending information between JavaScript and R, and then implemented this in the package.

Takeaways from a structured LLM Learning session

What made this work wasn’t that ChatGPT giving me the right code but that it helped me ask the right questions. It acted like a more senior engineer, talking me through unfamiliar concepts, helping me debug my code, and nudging me towards the right mental model.

Step 1: Convert the scripts into an R package

I’m a big fan of writing code as R packages wherever possible; it provides instant structure and reminds me to do important fundamentals such as documenting functions, and writing modular pieces instead of long scripts.

I moved the app code into ./R/app.R and separated out the other functions into a file ./R/prompts.R.

I also moved the CSS for the app into the package’s inst dir and created a subdirectory in inst to store my prompts.

Step 2: Saving the prompts in their own file

Next, was where to save the prompts. Originally these were saved as variables, but in the ellmer vignette on prompt design, it’s recommended that these are saved in ./inst/prompts/ with one file per prompt.

The ellmer vignette recommends saving prompts as markdown files as they’re both human-readable and LLM-readable, and so I saved my main prompt as shown below:

Create me social media posts for each of these platforms: {{platforms}}, to promote the blog post below.

* create {{n}} posts per platform
* use a {{tone}} tone
* use hashtags: {{hashtags}}
* use emojis? {{emojis}}

# Blog post contents

{{post_contents}}

You’ll notice the use of placeholders in the prompt. This is because {ellmer} has a helpful function which can read the prompt and inject in variable values. So the last line of my get_prompt() function looks like this:

ellmer::interpolate_file(
    system.file("prompts", "prompt-main.md", package = "socialmediagen"),
    platforms = paste(platforms, collapse = ", "),
    n = n,
    tone = tone,
    hashtags = hashtags,
    emojis = emojis,
    post_contents = post_contents
)

Step 3: Creating platform-specific prompts

I realised that best-practices for social media posts vary from platform to platform, and I wanted to experiment with having different prompts depending on which platforms were selected by the user.

I first added the following to my main prompt:

# Platform-specific advice

Use the following advice to customise the output for individual platforms:

{{platform_specific_advice}}

I then created additional prompts tailored to each platform, for example, my LinkedIn prompt looks like this:

LinkedIn:
* Keep posts between 1,300 and 2,000 characters. 
* Use short sentences: Posts with sentences under 12 words perform better. 
* ask questions: Encourage comments by asking questions that prompt discussion. 
* give specific instructions: Ask readers to like your post or take another action. 
* use a compelling headline: Grab attention with your first line. 
* use 3 hashtags

I then saved all of these into the inst/prompts directory.

I then created a super-simple function to retrieve the relevant prompt:

#' Retrieve post-writing advice unique to specific platforms
#'
#' @param platforms Which platforms to get advice for
get_platform_specific_advice <- function(platforms){
  prompt_files <- paste0("prompt-", tolower(platforms), ".md")
  file_paths <- system.file("prompts", prompt_files, package = "socialmediagen")

  contents <- lapply(file_paths, readLines)
  paste(unlist(contents), collapse = "\n")
}

Finally, I updated my get_prompt() function to incorporate these changes:

#' Construct LLM prompt
#'
#' Construct a LLM prompt based on user input
#'
#' @param blog_link URL of source material
#' @param platforms Social media platform to create prompts for
#' @param n Number of prompts to create for each platform
#' @param emojis Use emojis in post?
#' @param tone Desired tone of the post
#' @param hashtags Hashtags to include in the post
#' @importFrom ellmer interpolate_file
get_prompt <- function(blog_link, platforms, n, emojis, tone, hashtags) {
  # retrieve post contents from GitHub
  post_contents <- fetch_github_markdown(blog_link)

  platform_specific_advice <- get_platform_specific_advice(platforms)

  # combine components
  ellmer::interpolate_file(
    system.file("prompts", "prompt-main.md", package = "socialmediagen"),
    platforms = paste(platforms, collapse = ", "),
    n = n,
    tone = tone,
    hashtags = hashtags,
    emojis = emojis,
    post_contents = post_contents,
    platform_specific_advice = platform_specific_advice
  )
}

The resulting app

My app now is able to create platform-specific content and is much better organised than it was before. I think that updating the prompt was useful - check out the example below of the content generated before and after. The LinkedIn post now it has a strong call-to-action and asks the user a question.

📢 Calling all data enthusiasts! 📢

Thinking about speaking at Posit Conf 2025 but feeling a bit 😬 about the video abstract? Don't sweat it! This year the abstract just needs to be a short video!

We've compiled some top tips from past speakers to help you nail that one-minute pitch! 🚀

➡️ Learn how to structure your video, what tech to use, and how to inject some of YOUR personality into it! (Humor and creativity are welcome!)

Plus, we're offering feedback on drafts! 📝

[Link to Blog Post]

#rstats

**Headline:** Nervous About Your Posit Conf Video Abstract? 😩 Don't Be!

Thinking of speaking at #PositConf2025 but dreading the video abstract? You're not alone! It's just one minute to shine ✨.

We've got tips from past speakers Rachael Dempsey, Shannon Pileggi, and Nic Crane to help you nail it! From tech options (easy peasy phone cam to fancy OBS Studio) to structuring your pitch, we've got you covered.

**Blog Post Highlights:**

*   Simple tech options
*   Easy structure for your video
*   How to get feedback

Ready to record? Check out our tips and let your brilliance shine. 💡
[Link to blog post]

What's your biggest video recording fear? Share in the comments! 👇

#rstats #rladies #positconf

Reflecting on LLM-based apps

I enjoyed building this example, and on some levels, it doesn’t do anything particularly radical. The same results could be achieved by using the same prompts in a browser session and manually filling in the parameters. However, what I do like about this is that I have a specific place to store my prompts - in a GitHub repo - where I can iterate on them and track changes over time.

I also like that I now have a deployable artifact that I can share with others - I had mentioned to other members of the R-Ladies Global Team that I had been using LLMs to generate social media posts for promoting our news and blog posts, and this means I can easily share the link to the repo with the app, instead of having to share a prompt. Creating this as a distinct project encourages collaboration, whether that’s on which parameters we want to include in the app, or improving the quality of the prompts.

The Solution: A Shiny App Powered by AI

The app does the following:

• Takes a GitHub Markdown link as input.
• Lets users specify hashtags, the number of posts to generate, and whether or not to use emojis.
• Allows selection of tone (e.g., serious, playful, informative).
• Calls Google’s Gemini API via the ellmer package to generate social media posts promotinhg the blog post.

How It Works

The app is simple:

1. Enter the GitHub Markdown file link (or other text source).
2. Set your preferences—hashtags, emojis, tone, and how many posts to generate.
3. Click generate, and it spits out a bunch of posts.
4. Copy and paste into your social media scheduler.

Under the hood, it builds a prompt using the user’s inputs and calls the Gemini API. The response is formatted into platform-appropriate posts, making it much easier to schedule across different social networks.

Deployment Considerations

Right now, the app runs locally, and users need to enter their own Gemini API key as an evironment variable. This prevents abuse of a shared key, but it also makes the app a bit more DIY. I’m considering deploying it on Posit Connect Cloud or a similar service in the future. The trick will be allowing users to bring their own API keys while keeping things secure.

Another thought: supporting multiple AI models so users can plug in API keys for different services like OpenAI’s ChatGPT or Anthropic’s Claude.

Future Ideas

Some things I’d love to add:

• More refined prompts tailored to different platforms (e.g., longer for LinkedIn, punchier for Bluesky).
• Suggested images to go with posts.

Try It Yourself!

Want to give it a go? You can find the repo here: https://github.com/thisisnic/socialmediagen

Wrapping Up

This was a fun little side project that turned out to be actually useful - I’m going to be using it to promote this blog post!

I’d love to hear if you have any ideas for improvements—hit me up if you try it out!

How it all works

The repo itself it pretty simple in structure - it contains the GitHub Actions workflow, and a folder containing the changelogs for the packages I’m interested in.

# tree
.github
└── workflows
    └── compare_hashes.yml
changelogs
├── data.table-NEWS.md
├── dbplyr-NEWS.md
├── dplyr-NEWS.md
├── dtplyr-NEWS.md
├── duckdb-NEWS.md
├── duckplyr-NEWS.md
├── lubridate-NEWS.md
├── r-polars-NEWS.md
├── stringr-NEWS.md
└── tidypolars-NEWS.md

The workflow is triggered every day at 1am UTC, and runs a script that compares the hashes of the changelogs in my repo with the hashes of the changelogs in the package repos. If there’s a difference, the new changelog is saved to my repo and it sends me an email.

The GitHub Actions Workflow

In the next sections, I’ll walk through the GitHub Actions workflow, step by step, explaining what each bit does.

name: Check for package updates

on:
  schedule:
    # * is a special character in YAML so you have to quote this string
    - cron:  '00 1 * * *'

jobs:
  compare-hashes:
    name: ${{matrix.package.name}} updates
    runs-on: ubuntu-latest
    permissions:
      contents: write
    strategy:
        matrix:
          package: 
            [
                { name: dbplyr, file: dbplyr-NEWS.md, url: 'https://raw.githubusercontent.com/tidyverse/dbplyr/main/NEWS.md' }, 
                { name: lubridate, file: lubridate-NEWS.md, url: 'https://raw.githubusercontent.com/tidyverse/lubridate/main/NEWS.md' },
                { name: dplyr, file: dplyr-NEWS.md, url: 'https://raw.githubusercontent.com/tidyverse/dplyr/main/NEWS.md'}, 
                { name: data.table, file: data.table-NEWS.md, url: 'https://raw.githubusercontent.com/Rdatatable/data.table/master/NEWS.md'},
                { name: dtplyr, file: dtplyr-NEWS.md, url: 'https://raw.githubusercontent.com/tidyverse/dtplyr/main/NEWS.md'},
                { name: duckdb-r, file: duckdb-r-NEWS.md, url: 'https://raw.githubusercontent.com/duckdb/duckdb-r/main/NEWS.md'},
                { name: r-polars, file: r-polars-NEWS.md, url: 'https://raw.githubusercontent.com/pola-rs/r-polars/main/NEWS.md'},
                { name: stringr, file: stringr-NEWS.md, url: 'https://raw.githubusercontent.com/tidyverse/stringr/main/NEWS.md'},
                { name: duckplyr, file: duckplyr-NEWS.md, url: 'https://raw.githubusercontent.com/duckdblabs/duckplyr/main/NEWS.md'},
                { name: tidypolars, file: tidypolars-NEWS.md, url: 'https://raw.githubusercontent.com/etiennebacher/tidypolars/main/NEWS.md'},
            ]

    steps:
        - name: Checkout code
          uses: actions/checkout@v2
        - name: Get local file hash
          id: local-hash
          run: echo "local_hash=$(md5sum changelogs/${{ matrix.package.file }} | cut -d ' ' -f 1)" >> $GITHUB_OUTPUT

        - name: Get remote file
          id: remote-file
          run: |
                  mkdir tmp
                  curl -s ${{ matrix.package.url }} > tmp/${{ matrix.package.file }}#
        - name: Get remote file hash
          id: remote-hash
          run: echo "remote_hash=$(md5sum tmp/${{ matrix.package.file }} | cut -d ' ' -f 1)" >> $GITHUB_OUTPUT

        - name: Update changed files
          if: ${{ steps.local-hash.outputs.local_hash != steps.remote-hash.outputs.remote_hash }}
          run: |
            echo "Hashes do not match!"   
            git config --global user.email "github-actions[bot]@users.noreply.github.com"
            git config --global user.name "GHA Bot"
            cp tmp/${{ matrix.package.file }} changelogs/${{ matrix.package.file }}
            git add changelogs/${{ matrix.package.file }}
            git pull --ff-only
            git commit -m "Update ${{matrix.package.name}} changelog"
            git push

        - name: Send email notification
          if: ${{ steps.local-hash.outputs.local_hash != steps.remote-hash.outputs.remote_hash }}
          uses: dawidd6/action-send-mail@v3
          with:
            server_address: smtp.gmail.com
            server_port: 465
            username: ${{ secrets.MAIL_USERNAME }}
            password: ${{ secrets.MAIL_PASSWORD }}
            subject: "${{matrix.package.name}} update"
            body: "${{matrix.package.name}} has been updated! Please check the changelog at ${{matrix.package.url}}."
            to: ${{ secrets.MAIL_RECIPIENT }}
            from: ${{ secrets.MAIL_USERNAME}}

Conclusion

I really enjoyed working on this and learning more about GitHub Actions. This has proved to be a useful tool, though there are a few improvements that could be made:

• some packages update their changelog more frequently than others and so some of the updates I get feel a bit spammy. I could fix this by running my CRON job on a weekly rather than daily schedule.
• I don’t use this as much as I anticipated because some changes are really minor, and I tend to skim them and not pay too much attention. Again, a different CRON frequency could probably help here.
• I’m more interested in some packages than others. {dplyr}, {lubridate}, and {stringr} are the most important, whereas others are just a “nice to have” here. I could separate these out into different jobs, and run them on different schedules.

Anyway, I’d love to hear your thoughts - how do you keep up to date with changes in R packages? Do you have any suggestions for improvements to this workflow? Let me know! Get in touch via Mastodon or LinkedIn.

The tyranny of “should”

I’ve spent much of this year suffering from the tyranny of “should” regarding my own knowledge and abilities, in a way that has been verging on ridiculous. A lot of it is the perfect storm of adjusting to new responsibility, the deeply unhelpful way in which my self-image when I was younger was built up around being “smart” and “knowing things”, and a knack for punching above my weight by being good at figuring out just enough about a topic to get shit done. I don’t know every single detail of object orientation in R, but I know enough to be able to create new bindings between C++ objects and R6 classes in the arrow R package and expose their methods via fields and active bindings. Despite having looked it up repeatedly over the past few years, I still cannot recall the difference between how things work with static and shared libraries, but I did manage to fix a bug requiring me to bump the version of the C++ date library that Arrow vendors, via inspecting old pull requests and helpful feedback I received on the PR I submitted.

It came to a head at this year’s posit::conf, where I had the privilege of co-teaching a workshop about Arrow with the wonderful Stephanie Hazlitt. We spent months preparing materials and practicing teaching to our laptop screens, but when the event itself rolled around, teaching it to a room just shy of 60 people was a different experience entirely. It was intimidating being in front of a room of eager learners, and any jokey asides I was usually capable of making went out of the window, and I just taught the materials fairly plainly, stumbling over my words a bit, and finding the moments of silence in the room as I was teaching pretty awful. Subsequent feedback from folks I trust enough to deliver at least some honesty alongside their validation has led me to conclude that the only person who really noticed or cared about this was me. I was doing my old trick of snatching defeat from the jaws of victory, and realised later that my self-expectations were totally unreasonable. While I used to be able to absolutely smash through delivering a well-practiced 2-day “introduction to R” course, and delighted in engaging the folks in the room and showing them what they (and R!) were capable of, it’s 5 years since I last did that on the regular. This was the first time these materials had seen the light of day, and at no less than posit::conf, daunting enough on its own. And honestly, this is all ego. We ran a good workshop, people were engaged, and we wrote some quality reusable materials. Yes, there’s room for improvement, but overall it was solid. Delivery of content is something I can (and will) work on. I’ve applied for The Carpentries instructor training, and have plans to get back into the swing of teaching by running workshops (on Arrow and other topics) at user events.

The power of “I don’t know but I can find out”

The second day of posit::conf was spent being a teaching assistant for the fantastic Andy Teucher, who was teaching “Fundamentals of Package Development”. I managed to escape a lot of this “should” nonsense here, so when one of the course attendees asked me about where to store CSV files to use in unit tests, I happily admitted I wasn’t sure but could find out, and together we had a look in “Writing R Packages”, where I couldn’t find the information, and then suggested the approach I usually take when working on arrow - look at what folks who often define best practices do in the packages that they maintain. We took a look at readr, found the files in the testthat directory, and concluded that unless there’s reason to want to do anything more complicated (like make those datasets available to end users), then that approach is fine. Not knowing the answer instantly wasn’t a hindrance to working out the necessary solution. Learning how to say “I don’t know, but let’s find out together” is one of the most important things I learned when I was first teaching people how to use R. A teacher doesn’t have to know everything, but should be able to help a learner find the information they need.

The power of community

The rest of the conference week was pretty awesome. I was in the middle of a CRAN resubmission (well, actually a re-re-resubmission, but let’s not go there) which had got a bit hairy, and I needed some help tracking down the source of the bug. I spent some time briefly engaging in a bit of pair programming with Jonathan Keane and Neal Richardson, who’ve both been involved with Arrow for a lot longer than me, and I’m always taken aback by just how quickly they both get to the source of a problem any time I ask for help. Part of my “knowledge anxiety” manifests as me sometimes taking longer than I would like to process things when communicating verbally about technical topics, which leads to me getting really frustrated when I’m just not “getting” things in the moment that I know later will seem pretty obvious and uncomplicated. This happened when we were looking at the arrow bug, but I had this sudden moment of clarity upon realising…nobody cared except me. Jon and Neal are great collaborators and are always happy to explain things again if there’s time, and aren’t going to judge me for it.

I also realised that there is no universal measure of what I should know or where I should be at. While I always want to know more about how things work, I don’t have to know everything and it’s a pretty unrealistic expectation for me to have of myself; sure I can (and often do!) go read a whole load of the codebase and related concepts, but that’ll never be the same as also leaning on other people’s years of experience and knowledge. The whole point of being part of a community is to be able to share both overlapping and divergent knowledge.

Progress is power

I don’t think there is a universal solution to this problem of feeling like you “should” know more, but all I know is what has worked for me. Public work is important - social media, blog posts, talks, workshops, whatever. It can be scaled to wherever you’re at. There’s always something you’ll be surprised that you know but others don’t. And it creates a lovely feedback loop whereby a little bit of external validation can go a long way.

Learning is also important - I did the first half of the Udacity C++ Nanodegree this year, and while a lot of what it taught me was that I never want to be a full-blown C++ developer (I’d rather make weird buildings and spaceships out of existing Lego pieces than become a polymer scientist just to create custom bricks), realising that I could learn C++ at that level if I really wanted to, was invaluable. Part of the course involved getting acquainted with cmake, and whilst I can’t claim expertise there, dabbling in a bit of the whats and whys helped make many related Arrow project issues seem less mystical.

Reassurance and validation are good and well, but in my experience, having tangible proof of the things that I do know is more effective.

“Fake it til you make it” is deeply problematic

The commonly received advice is “fake it ‘til you make it”. Pretend you feel like you belong and are confident, until that becomes the case. I’ve spent a long time attempting this, and the problem is that it doesn’t actually work, because it doesn’t address the underlying issue. It’s only since I started to own the fact that I don’t feel comfortable in every environment or domain and working out what I need to do to feel more comfortable that I’ve felt my confidence growing.

In the past month, I’ve been a lot more open about how I’ve been feeling around this. It’s been tricky as I’ve been scared that my vulnerability just looks weakness that I shouldn’t show around other people, or just come off as moaning, but it’s had massive benefits. During the first couple of days of posit::conf this year, I suffered the worst ongoing anxiety I’ve had all year - I had a constant knot in my chest - but despite that managed to have a good time as everyone was very accepting. I can say without a doubt that pretending to be fine would have made things infinitely worse.

There was another point during the conference when I casually mentioning that I was feeling quite hopeful about a potential opportunity but not entirely sure if it was in the bag, as I suspected I might be up against someone whose background I find impressive. It was kindly pointed out to me that each of us bring different things to the table, a comment which has since set off a chain reaction of me starting to appreciate what I can do rather than what I can’t.

The cure for imposter syndrome

The most important thing I learned this year is that the cure for imposter syndrome isn’t persuading yourself that you aren’t just 3 goats in a raincoat pretending to be a person, but instead surrounding yourself with folks who wouldn’t really care if you were anyway because goats are cool and that’s a pretty awesome feat of acrobatics and balancing.

How does type inference work in readr?

Since readr version 2.0.0 (released in July 2020), there was a significant overhaul of the underlying code, which subsequently depended on the vroom package.

The type inference is done by a C++ function in vroom called guess_type__() which guesses types in the following order:

• Does the column contain 0 rows? If yes, return “character”

• Are all values missing? If yes, return “logical”

• Tries to parse column to each of these formats and returns the first one it successfully parses:

  • Logical
  • Integer (though the default is to not look for these)
  • Double
  • Number (a special type which can remove characters from strings representing numbers and then convert them to doubles)
  • Time
  • Date
  • Datetime
  • Character

The ordering above in the parsing bullet point goes from most to least strict in terms of the conditions which have to be met to successfully parse an input as that data type. For example, for a column to be of logical type, it can only contain a small subset of values representing true (T, t, True, TRUE, true), false (F, f, False, FALSE, false) or NA, which is why this is the most strict type, but all of the other types could be read in as character data, which is the least strict and why this is last in the order.

How does type inference work in arrow?

In arrow, read_csv_arrow() handles CSV reading, and much of its interface has been designed to closely follow the excellent APIs of readr::read_csv() and vroom::vroom(). The intention here is that users can use parameter names they’re familiar with from the aforementioned readers when using arrow, and get the same results. The underlying code is pretty different though.

In addition, Arrow has a different set of possible data types compared to R; see the Arrow docs for more information about the mapping between R data types and Arrow types.

In the Arrow docs, we can see that types are inferred in this order:

• Null
• Int64
• Boolean
• Date32
• Timestamp (with seconds unit)
• Float64
• Dictionary<String> (if ConvertOptions::auto_dict_encode is true)
• Dictionary<Binary> (if ConvertOptions::auto_dict_encode is true)
• String
• Binary

Note that if you use arrow::read_csv_arrow() with parameter as_data_frame = TRUE (the default), the Arrow data types are then converted to R data types.

simple_data <- data.frame(x = c(1, 2, 3), y = c("a", "b", "c"), z = c(1.1, 2.2, 3.3))

readr::write_csv(simple_data, "simple_data.csv")

# columns are arrow's int64, string, and double (aka float64) types
arrow::read_csv_arrow("simple_data.csv", as_data_frame = FALSE)

## Table
## 3 rows x 3 columns
## $x <int64>
## $y <string>
## $z <double>

# columns converted to R's integer, character, and double types
arrow::read_csv_arrow("simple_data.csv", as_data_frame = TRUE)

## # A tibble: 3 x 3
##       x y         z
##   <int> <chr> <dbl>
## 1     1 a       1.1
## 2     2 b       2.2
## 3     3 c       3.3

What are the main differences between readr and arrow type inference?

Although there appear to be quite a few differences between the order of type inference when comparing arrow and readr, in practice, this doesn’t have much effect. Type inference for logical/boolean and integer values are the opposite way round, but given that the underlying data that translates into these types looks very different, they’re not going to be mixed up. The biggest differences come from custom behaviours which are specific to readr and arrow; I’ve outlined them below.

int_or_dbl <- data.frame(
  x = c(1L, 2L, 3L)
)

readr::write_csv(int_or_dbl, "int_or_dbl.csv")

readLines("int_or_dbl.csv")

## [1] "x" "1" "2" "3"

# double
readr::read_csv("int_or_dbl.csv")

## Rows: 3 Columns: 1
## -- Column specification --------------------------------------------------------
## Delimiter: ","
## dbl (1): x
##
## i Use `spec()` to retrieve the full column specification for this data.
## i Specify the column types or set `show_col_types = FALSE` to quiet this message.

## # A tibble: 3 x 1
##       x
##   <dbl>
## 1     1
## 2     2
## 3     3

# integer via inference
readr::read_csv("int_or_dbl.csv", col_types = list(.default = col_character())) %>%
  type_convert(guess_integer = TRUE)

##
## -- Column specification --------------------------------------------------------
## cols(
##   x = col_integer()
## )

## # A tibble: 3 x 1
##       x
##   <int>
## 1     1
## 2     2
## 3     3

# integer via specification
readr::read_csv("int_or_dbl.csv", col_types = list(col_integer()))

## # A tibble: 3 x 1
##       x
##   <int>
## 1     1
## 2     2
## 3     3

# integer via inference
arrow::read_csv_arrow("int_or_dbl.csv")

## # A tibble: 3 x 1
##       x
##   <int>
## 1     1
## 2     2
## 3     3

sixty_four <- data.frame(x = 2^31 - 1, y = 2^31)

readr::write_csv(sixty_four, "sixty_four.csv")

# doubles by default
readr::read_csv("sixty_four.csv")

## Rows: 1 Columns: 2
## -- Column specification --------------------------------------------------------
## Delimiter: ","
## dbl (2): x, y
##
## i Use `spec()` to retrieve the full column specification for this data.
## i Specify the column types or set `show_col_types = FALSE` to quiet this message.

## # A tibble: 1 x 2
##            x          y
##        <dbl>      <dbl>
## 1 2147483647 2147483648

# 32 bit integer or double depending on value size
readr::read_csv("sixty_four.csv", col_types = list(.default = col_character())) %>%
  type_convert(guess_integer = TRUE)

##
## -- Column specification --------------------------------------------------------
## cols(
##   x = col_integer(),
##   y = col_double()
## )

## # A tibble: 1 x 2
##            x          y
##        <int>      <dbl>
## 1 2147483647 2147483648

# integers by specification
readr::read_csv(
  "sixty_four.csv",
  col_types = list(x = col_integer(), y = vroom::col_big_integer())
)

## # A tibble: 1 x 2
##            x          y
##        <int>    <int64>
## 1 2147483647 2147483648

# integers by inference
arrow::read_csv_arrow("sixty_four.csv")

## # A tibble: 1 x 2
##            x          y
##        <int>    <int64>
## 1 2147483647 2147483648

number_type <- data.frame(
  x = c("1,000", "1,250")
)

readr::write_csv(number_type, "number_type.csv")

# double type, but parsed in as number in column spec shown below
readr::read_csv("number_type.csv")

## Rows: 2 Columns: 1
## -- Column specification --------------------------------------------------------
## Delimiter: ","
## num (1): x
##
## i Use `spec()` to retrieve the full column specification for this data.
## i Specify the column types or set `show_col_types = FALSE` to quiet this message.

## # A tibble: 2 x 1
##       x
##   <dbl>
## 1  1000
## 2  1250

# read in as character data in Arrow
arrow::read_csv_arrow("number_type.csv")

## # A tibble: 2 x 1
##   x
##   <chr>
## 1 1,000
## 2 1,250

dict_type <- data.frame(
  x = c("yes", "no", "yes", "no")
)

readr::write_csv(dict_type, "dict_type.csv")

# character data
readr::read_csv("dict_type.csv")

## Rows: 4 Columns: 1
## -- Column specification --------------------------------------------------------
## Delimiter: ","
## chr (1): x
##
## i Use `spec()` to retrieve the full column specification for this data.
## i Specify the column types or set `show_col_types = FALSE` to quiet this message.

## # A tibble: 4 x 1
##   x
##   <chr>
## 1 yes
## 2 no
## 3 yes
## 4 no

# factor data
readr::read_csv("dict_type.csv", col_types = list(x = col_factor()))

## # A tibble: 4 x 1
##   x
##   <fct>
## 1 yes
## 2 no
## 3 yes
## 4 no

# set up the option. there's an open ticket to make this code a bit nicer to read.
auto_dict_option <- arrow::CsvConvertOptions$create(auto_dict_encode = TRUE)
arrow::read_csv_arrow("dict_type.csv", convert_options = auto_dict_option)

## # A tibble: 4 x 1
##   x
##   <fct>
## 1 yes
## 2 no
## 3 yes
## 4 no

alternative_true_false <- arrow::CsvConvertOptions$create(
  false_values = "no", true_values = "yes"
)
arrow::read_csv_arrow("dict_type.csv", convert_options = alternative_true_false)

## # A tibble: 4 x 1
##   x
##   <lgl>
## 1 TRUE
## 2 FALSE
## 3 TRUE
## 4 FALSE

Using schemas for manual control of data types

Although relying on the reader itself to guess your column types can work well, what if you want more precise control?

In readr, you can use the col_types parameter to specify column types. You can use the same parameter in arrow to use R type specifications.

given_types <- data.frame(x = c(1, 2, 3), y = c(4, 5, 6))

readr::write_csv(given_types, "given_types.csv")

readr::read_csv("given_types.csv", col_types = list(col_integer(), col_double()))

## # A tibble: 3 x 2
##       x     y
##   <int> <dbl>
## 1     1     4
## 2     2     5
## 3     3     6

You can also use this shortcode specification. Here, “i” means integer and “d” means double.

readr::read_csv("given_types.csv", col_types = "id")

## # A tibble: 3 x 2
##       x     y
##   <int> <dbl>
## 1     1     4
## 2     2     5
## 3     3     6

In arrow you can use the shortcodes (though not the col_*() functions), but you must specify the column names.

We skip the first row as our data has a header row - this is the same behaviour as when we use both names and types in readr::read_csv() which then assumes that the header row is data if we don’t skip it.

arrow::read_csv_arrow("given_types.csv", col_names = c("x", "y"), col_types = "id", skip = 1)

## # A tibble: 3 x 2
##       x     y
##   <int> <dbl>
## 1     1     4
## 2     2     5
## 3     3     6

What if you want to use Arrow types instead of R types though? In this case, you need to use a schema. I won’t go into detail here, but in short, schemas are lists of fields, each of which contain a field name and a data type. You can specify a schema like this:

# this gives the same result as before - because our Arrow data has been converted to the relevant R type
arrow::read_csv_arrow("given_types.csv", schema = schema(x = int8(), y = float32()), skip = 1)

## # A tibble: 3 x 2
##       x     y
##   <int> <dbl>
## 1     1     4
## 2     2     5
## 3     3     6

# BUT, if you don't read it in as a data frame you'll see the Arrow type
arrow::read_csv_arrow("given_types.csv", schema = schema(x = int8(), y = float32()), skip = 1, as_data_frame = FALSE)

## Table
## 3 rows x 2 columns
## $x <int8>
## $y <float>

Parquet

An alternative approach is to use Parquet format, which stores the data types along with the data. This means that if you’re sharing your data with others, you don’t need to worry about it being read in as the wrong data types. In a follow-up post I’ll explore the Parquet format and compare management of data types in CSVs and Parquet.

Further Reading

If you want a much more detailed discussion of Arrow data types, see this excellent blog post by Danielle Navarro.

Thanks

Huge thanks to everyone who helped review and tweak this blog post, and special thanks to Jenny Bryan who gave some really helpful feedback on the content on readr/vroom!

What is Acero?

Acero is an Arrow-native query execution engine, developed as part of the Arrow C++ / R libraries.

What is a query execution engine?

A query execution engine is a piece of software which allows users to execute queries against a data source. It typically has multiple steps, which could include things like taking the query and parsing it into an algebraic format, re-ordering and optimising the query in order to run it in the most efficient way, and actually running the query.

What is an ExecPlan?

Queries to be run on Acero are specified as execution plans aka ExecPlans. These are directed graphs which express what operations need to take place via nodes - called ExecNodes in this case. The graph for the ExecPlan in the code example earlier in this post looks like this:

TableSourceNode -> FilterNode -> ProjectNode -> SinkNode

The ExecNodes in the plan above are:

• TableSourceNode - used to input the data

• FilterNode - where we apply our filter to only retain rows where the value of mpg is greater than 20

• ProjectNode - this then specifies which columns we want, including the new column wt_kg as a compute expression

• SinkNode - this provides the output

The problem

I was excited to work on this feature as its importance is clear to me from my consulting days; I’ve worked with many organisations who have had to lock down versions of R packages they allow employees to install, and so upgrading to the latest version isn’t an option. Arrow is growing and developing as a project, both in terms of internal functionality and nudges to improve package UX, and so the latest docs version may be inaccurate if users are working with a version of the package from 1 year ago, given we do major releases every three months.

Does this already exist in pkgdown?

It should be noted that pkgdown does already support having documentation for dev and release versions of packages, which is sufficient for most cases, but it doesn’t currently support more versions than that. This has been discussed in an issue, and I’d considered trying to do something which I could submit as a PR to pkgdown, but it’s complicated by having to make a number of design decisions which would work at a general level (and not just specific to Arrow), and, frankly, not having the resources to invest the time and effort required to come up with this kind of solution.

Instead, at the suggestion of a colleague, I opted for a simpler solution - using custom JavaScript to override the version badge on the page and replace it with a dropdown selector.

The solution

Arrow already has the CI set up to deploy different versions of the docs to specific URLs - the task here is just to point pkgdown to the right place. The docs are deployed to the following URLs:

• https://arrow.apache.org/docs/r/ - the released version
• https://arrow.apache.org/docs/dev/r/ - the dev version
• https://arrow.apache.org/docs/[arrow release version number]/r/ - a previous version

When creating a pkgdown site, you can display the package version in the navbar. Here’s how this looks on an older version of the Arrow R package docs:

The plan was to use jQuery to grab this element from the page and replace it with a dropdown that has the current version selected but allows the user to navigate to another version.

[
    {
        "name": "7.0.0.9000 (dev)",
        "version": "dev/"
    },
    {
        "name": "7.0.0 (release)",
        "version": ""
    },
    {
        "name": "6.0.1",
        "version": "6.0/"
    },
    {
        "name": "5.0.0",
        "version": "5.0/"
    },
    {
        "name": "4.0.1",
        "version": "4.0/"
    },
    {
        "name": "3.0.0",
        "version": "3.0/"
    },
    {
        "name": "2.0.0",
        "version": "2.0/"
    },
    {
        "name": "1.0.1",
        "version": "1.0/"
    }
]

$(document).ready(function () {

  /**
   * This replaces the package version number in the docs with a
   * dropdown where you can select the version of the docs to view.
   */

     // Get the start of the path which includes the version number or "dev"
     // where applicable and add the "/docs/" suffix
    $pathStart = function(){
      return window.location.origin + "/docs/";
    }

    // Get the end of the path after the version number or "dev" if present
    $pathEnd  = function(){
      var current_path = window.location.pathname;
      return current_path.match("(?<=\/r).*");
    }

    // Load JSON file mapping between docs version and R package version
    $.getJSON("https://arrow.apache.org/docs/r/versions.json", function( data ) {
      // get the current page's version number:
      var displayed_version = $('.version').text();
      // Create a dropdown selector and add the appropriate attributes
      const sel = document.createElement("select");
      sel.name = "version-selector";
      sel.id = "version-selector";
      sel.classList.add("navbar-default");
      // When the selected value is changed, take the user to the version
      // of the page they are browsing in the selected version
      sel.onchange = check_page_exists_and_redirect;

      // For each of the items in the JSON object (name/version pairs)
      $.each( data, function( key, val ) {
        // Add a new option to the dropdown selector
        const opt = document.createElement("option");
        // Set the path based on the 'version' field
        opt.value = $pathStart() + val.version + "r" + $pathEnd();
        // Set the currently selected item based on the major and minor version numbers
        opt.selected = val.name.match("[0-9.]*")[0] === displayed_version;
        // Set the displayed text based on the 'name' field
        opt.text = val.name;
        // Add to the selector
        sel.append(opt);
      });

      // Replace the HTML "version" component with the new selector
      $("span.version").replaceWith(sel);
    });
});

function check_page_exists_and_redirect(event) {

    /**
       * When a user uses the version dropdown in the docs, check if the page
       * they are currently browsing exists in that version of the docs.
       * If yes, take them there; if no, take them to the main docs page.
       */

    const path_to_try = event.target.value;

    const base_path = path_to_try.match("(.*\/r\/)?")[0];
    let tryUrl = path_to_try;
    $.ajax({
        type: 'HEAD',
        url: tryUrl,
        success: function() {
            location.href = tryUrl;
        }
    }).fail(function() {
        location.href = base_path;
    });
    return false;
}

Testing

I considered testing this locally, but it was tricky because the code for breaking up the URL didn’t quite work properly when running this on localhost. I could have tried to fix that, but I’d also have to simulate having multiple versions of deployed docs, and ultimately it was much simpler to test on the live docs instead.

$(window).bind("pageshow", function(event) {
  if (event.originalEvent.persisted) {
    window.location.reload()
  }
});

$pathEnd  = function(){
  var current_path = window.location.pathname;
  return current_path.match("(?<=\/r).*");
}

$pathEnd  = function(){
  var current_path = window.location.pathname;
  // Can't do this via a positive look-behind or we lose Safari compatibility
  return current_path.match("\/r.*")[0].substr(2);
}

Conclusion

Awesome, thanks for sticking with me as I walked through that. This was honestly a bit of a PITA to do as I’m not the best JS developer, and the bugs were a pain to solve without being able to easily replicate them with my browsers/OS (I’m sure there are tools out there to do it but honestly just having a quick call with a colleague was the simplest solution!) I would have loved to have done something where I ended up submitting a PR to pkgdown to make this a more general feature, but this solution is super-custom and isn’t set up that way. That said, this was a super interesting problem to solve, and it was good fun working out how to fit all the pieces together!

Introduction

Let’s imagine I work in the HR department of the UK’s number 1 employer of cats. Our employees are available for modelling, snuggles, and avian assassination. Our individual offices keep records of new employees, and send them to me via email so I can collate them. Our process could be better, but it works for now.

Basic data validation

When I get a new set of records, I want to do some analysis, and because I’m awesome, I want to do it in R. Unfortunately our regional office input data manually, which is subject to human error. I’ve decided I want to write some R code which automates data import and validation for me.

I know that Dave in the Birmingham office often inputs the cat’s ages wrong - too little caffeine in the mornings really throws him off his game - but given the fact that the world’s oldest cat was 38, we can safely say that if the data shows an age older than that it’s either an input error or cause to call the Guinness Book of World Records.

Here’s my initial function with import and basic validation!

library(readr)
library(dplyr, warn.conflicts = FALSE)
library(rlang)

# function with basic validation
import_cat_data <- function(file){

  # Column types are always the same so I may as well specify them here
  data <- readr::read_csv(
    file,
    col_types = cols(
      col_double(),
      col_integer(),
      col_double()
    )
  )

  # Validation
  if (any(data$age > 38)) {
    rlang::abort(
      c(
        "Values in `age` must be 38 or less",
        paste("Invalid values detected:", paste0(data$age[data$age > 38], collapse = ", "))
      )
    )
  }

  data
}

So how does this look if I have perfectly good data?

# Create some example data and write to a CSV dile
cats <- tibble::tibble(age = c(5, 0.5, 13), paws = rep(4, 3), teeth = c(30, 26, 26))
readr::write_csv(cats, file = "cats.csv")

# Import data
import_cat_data("cats.csv")

## # A tibble: 3 x 3
##     age  paws teeth
##   <dbl> <int> <dbl>
## 1   5       4    30
## 2   0.5     4    26
## 3  13       4    26

And how does this look when I run it on Dave’s dodgy data?

dave_data <- tibble::tibble(age = c(10, 200), paws = c(3, 4), teeth = c(30, 30))
readr::write_csv(dave_data, file = "dave_data.csv")

# Triggers the error
import_cat_data("dave_data.csv")

## Error in `import_cat_data()`:
## ! Values in `age` must be 38 or less
## * Invalid values detected: 200

Custom warning handling

OK, so we’ve covered Dave’s dodgy data, but what other problems do I have in my pipeline? Sometimes Anja in our Wigan branch will send me .tsv. files instead of .csv. I’m not interested in detecting the file type - I just want an error message. So, what does it look like when I try to import that data?

# Set up tsv file saved as a csv
even_more_cats <- tibble::tibble(age = c(7, 3), paws = c(4, 4), teeth = c(30, 28))
cats_tsv <- readr::write_tsv(even_more_cats, file = "even_more_cats.csv")

# Import the file
import_cat_data("even_more_cats.csv")

## Warning: Unnamed `col_types` should have the same length as `col_names`. Using
## smaller of the two.

## Warning: 2 parsing failures.
## row              col               expected   actual                 file
##   1 age  paws    teeth no trailing characters 7  4   30 'even_more_cats.csv'
##   2 age  paws    teeth no trailing characters 3  4   28 'even_more_cats.csv'

## Warning: Unknown or uninitialised column: `age`.

## # A tibble: 2 x 1
##   `age\tpaws\tteeth`
##                <dbl>
## 1                 NA
## 2                 NA

Urgh, this is kinda messy. I keep forgetting that it’s Anja’s dodgy tsv files which cause this, so I want to do 2 things here:

1. Promote the warning to an error
2. Give myself a little reminder about the cause of the error

I’ve now wrapped my data import stage in a tryCatch() so I can provide some custom behaviour if this warning appears, via another function I’ve written called handle_cats_import_warning().

import_cat_data <- function(file){

  tryCatch(
    data <- readr::read_csv(
      file,
      col_types = cols(
        col_double(),
        col_integer(),
        col_double()
      )
    ),
    warning = function(w){
      handle_cats_import_warning(w)
    }
  )

  if (any(data$age > 38)) {
    rlang::abort(
      c(
        "Values in `age` must be 38 or less",
        paste("Invalid values detected:", paste0(data$age[data$age > 38], collapse = ", "))
      )
    )
  }

  data
}

The warning helper is below. Basically, it extracts the message from the warning, and if this message matches the one I saw above, I append it with an extra little informational message as a hint about the possible cause. Then I call rlang::abort() to raise an error containing the content of the message. Remember, even if the warning is caused by something else, I still want it to error.

handle_cats_import_warning <- function(w){
  msg <- conditionMessage(w)
  if (grepl("Unnamed `col_types` should have the same length as `col_names`.", msg)) {
    msg <- c(
        msg,
        i = "Is the file you're importing a `.tsv`? Only `.csv' files are accepted."
    )
  }
  rlang::abort(msg)
}

So how does this look now?

import_cat_data("even_more_cats.csv")

## Error in `handle_cats_import_warning()`:
## ! Unnamed `col_types` should have the same length as `col_names`. Using smaller of the two.
## i Is the file you're importing a `.tsv`? Only `.csv' files are accepted.

It looks OK, but it’s not done yet - you might have noticed that the error is reported as coming from handle_cats_import_warning(). This isn’t quite right - that function is just the warning helper; I want to report the error as coming from import_cat_data(). So how do I do this? Let’s take a quick look at the source of the error - we’ll come back to this later.

rlang::last_error()

## <error/rlang_error>
## Error in `handle_cats_import_warning()`:
## ! Unnamed `col_types` should have the same length as `col_names`. Using smaller of the two.
## i Is the file you're importing a `.tsv`? Only `.csv' files are accepted.
## Backtrace:
##  1. global import_cat_data("even_more_cats.csv")
##  2. base::tryCatch(...)
##  3. base tryCatchList(expr, classes, parentenv, handlers)
##  4. base tryCatchOne(expr, names, parentenv, handlers[[1L]])
##  5. value[[3L]](cond)
##  6. global handle_cats_import_warning(w)
## Run `rlang::last_trace()` to see the full context.

We can see from the enumerated items on the backtrace that the function we want to show as the source of the error (import_cat_data()) is 5 items higher that the function currently being shown as the source of the error (handle_cats_import_warning()). So how do we change the reported error source?

Changing the calling environment

It basically comes down to using the call parameter when calling rlang::abort(). If you take a look at the docs, you’ll see it documented as:

The execution environment of a currently running function, e.g. call = caller_env(). The corresponding function call is retrieved and mentioned in error messages as the source of the error.

You only need to supply call when throwing a condition from a helper function which wouldn’t be relevant to mention in the message.

OK, perfect! So all I need to do is add another parameter to my warning helper function below - this means that wherever I’m calling it from, I can pass in information about the correct environment to report in the error message.

handle_cats_import_warning <- function(w, call){
  msg <- conditionMessage(w)
  if (grepl("Unnamed `col_types` should have the same length as `col_names`.", msg)) {
    msg <- c(
        msg,
        i = "Is the file you're importing a `.tsv`? Only `.csv' files are accepted."
    )
  }
  rlang::abort(msg, call = call)
}

Next, I need to update my call to tryCatch() to also incorporate this change, and I use rlang::caller_env() to specify the environment. The parameter n = 4 means “go back 4 callers”.

import_cat_data <- function(file){

  tryCatch(
    data <- readr::read_csv(
      file,
      col_types = cols(
        col_double(),
        col_integer(),
        col_double()
      )
    ),
    warning = function(w, call = caller_env(n = 4)){
      handle_cats_import_warning(w, call)
    }
  )

  # Validation

  if (any(data$age > 38)) {
    rlang::abort(
      c(
        "Values in `age` must be 38 or less",
        paste("Invalid values detected:", paste0(data$age[data$age > 38], collapse = ", "))
      )
    )
  }

  data
}

I discovered this number through trial and error, but after reading a bit more about the stack trace, I have a better idea about the reason Remember before when I said the function call we wanted to say was the error’s source was five items higher on the stack? In the warning handler above, we call caller_env() one place higher in the stack than handle_cats_import_warning(), and so 5 - 1 = 4.

So now let’s take a look - is our error message attributed to the right place in the call stack?

import_cat_data("even_more_cats.csv")

## Error in `import_cat_data()`:
## ! Unnamed `col_types` should have the same length as `col_names`. Using smaller of the two.
## i Is the file you're importing a `.tsv`? Only `.csv' files are accepted.

Yes, it is!

It was super interesting figuring out the details of this, and if you need to write code which involves error helpers and/or error chaining, I’d recommend that you check out this excellent rlang vignette on error chaining which covers even more things you can do around this.

Open-ended role

I’m doing a better job of adapting to the environment of total freedom within the role. My team has quarterly initiatives to complete, but otherwise, my job is just to “be an open-source maintainer” and do whatever that entails, with colleagues occasionally asking me to get involved with specific pieces of work. At first, this was deeply disorienting (and certainly I’m not the only one who feels this) - I want to be good at this and to be good at it, I need something to measure myself again, and super-explicit expectations and concrete standards for everything are crucial in this, right? Actually…no. Like everyone else, I have areas in which I’m stronger and weaker, and this environment allows me to play to my strengths, work on my weaknesses when it’s a good time to do so, and figure out where I fit in the wider team. If something isn’t getting done which falls within my remit, I’ll either figure it out myself or hear about it, and it’ll get sorted. If I’m doing something catastrophically wrong, I’m sure someone will let me know and help me work out how to fix it. Beyond that, I need to take ownership of asking for feedback, which is weirdly daunting - even from colleagues that I get on really well with - but learning to bite the bullet and do it is important.

On a practical level, I’ve found that keeping a spreadsheet of things I’m working on and things I intend to work on soon, with checklists of things I need to check each day, has been an effective and straightforward way of organising my time and priorities on a daily/weekly basis. And when I find the time, I plan on making something - perhaps a Shiny dashboard, Slack bot, or API - to help me keep track of answers to questions like “have any PRs been approved but not merged?”, “are there any new R PRs which haven’t yet been reviewed?” and “are there any new GitHub Issues without replies?”

Learning and documentation

It’s been a delight to find out that learning new things and producing documentation can be deeply intertwined processes, and I’ve noticed a three-stage process that seems to work for me.

1. Ingestion. This is where I write a ton of scruffy biro notes, draw diagrams, and sometimes even transcribe chunks of lectures verbatim when the topic is particularly far from my current knowledge. These usually get blu-tacked to my office wall to use as reference materials.
2. Consolidation. I take my notes from the ingestion stage, chuck away the irrelevant bits and condense them down to a smaller reference doc. This could be another piece of A4 stuck on the wall or a Google Doc. This is when I start to get excited that things are making a lot more sense.
3. Documentation. At some point, I realise that the thing I’ve learned is useful to someone else, and now I have to write it up properly. This is a weird bit of the process as I can’t tell you how long something is going to take or how I know when it’s done - it just kind of happens. I can’t say I love this bit once I’m three rounds of edits and reshuffling in, but once it’s done and I have some content, it’s pretty satisfying.

C++

When I first started, I felt like I wanted to learn everything, and very earnestly started picking up books on C++ and proclaiming I was going to learn the language. Since then, I’ve dabbled a bit but not taken it much further, and more recently, I’ve decided to change the scope of those ambitions. If programming in R and Python are like building things out of Lego bricks, C++ is more akin to becoming a polymer scientist and learning how to build the bricks. Worthwhile and interesting, sure, but not something I can get excited about in the same way. All that said, however, I’m still looking at learning “enough C++ to be useful in an R context”; I guess the equivalent of knowing enough about manufacturing to build my own custom bricks from time to time.

The SO non-incident

There was something that happened around month 5, which was really important at the time, and I’ve wanted to write about it. I’d replied to my second Arrow question on Stack Overflow - a little intimidating at the time, interacting in a different domain with different social norms compared to internally at Voltron Data or within the Apache Arrow project. Anyway, my (accurate and correct!) answer got deleted, and upon first glance, it looked like it had happened unfairly (which historically has disproportionately happened to people who aren’t your stereotypical dev). Some of my colleagues took this really seriously and were trying to see what they could do about it. Later, it transpired that this was not the case; I’d accidentally transgressed some rule relating to linking to an external source without pasting the relevant bit of content into my answer, and I submitted a new answer, which was not deleted. But, my colleagues’ initial reaction to what basically appeared to be discriminatory behaviour was…everything. I feel grateful to work with such excellent humans, and this whole incident made me realise it’s time to replace my mindset of “some people will treat you like this and best avoid those places/interactions if you don’t want to deal with that BS” with a new attitude of “you absolutely belong here, and people have got your back”.

Being an open-source maintainer

Working for Voltron Data and being a maintainer on Apache Arrow is one of the coolest jobs I’ve had in my life, if I’m honest, and I still have moments where I go “huh, I’m paid to work on open-source? Wow!” Don’t get me wrong, there are boring bits where I can spend more hours than I’d like tracking down an elusive bug than only shows up on a certain CI job or resolving some horrible merge conflicts after a rebase. But, these are balanced with things like working with new contributors on their first PR, writing documentation that turns implicit knowledge into explicit writing that can be shared, learning so so many new things, and submitting PRs for things that I personally care about on the project. My highlight reel so far includes:

• Submitting PRs to other projects - having actual problems I wanted to solve, even if it was just adding extra documentation or fixing typos (as opposed to trying to find things to do/fix as I was more invested in the idea of being a contributor), was really gratifying.
• Getting added to the list of authors on the arrow R package - I’d submitted a ton of PRs for that release but wasn’t expecting to get added to the author list, so it was a great surprise.
• Becoming a committer on the Apache Arrow project - being given the power to merge PRs was a little intimidating at first, but again, this privilege/responsibility was something I’d thought was further away, so it was a super exciting moment when it happened and it was nice to be recognised as “demonstrating a sustained commitment to the project”.
• Being heavily involved in publishing the first versions of the Apache Arrow cookbooks - I’m one of the main authors on the R cookbook, and whilst there’s a load of changes I’d like to see, getting the first version out was a real achievement.
• Updating the R package docs - I’m passionate about documentation and did a lot of work for this release adding diagrams and rephrasing the (already great) docs to try to make them simpler to understand and reach a wider audience of users.
• Publishing docreview on CRAN - I desperately need to make time to work on this side project, but getting the initial version of it together was instrumental in helping me work on the arrow docs and learn a few things about being a maintainer and starting new projects.
• Getting involved in a lot of CI work - our CI is pretty complex, and this is an area that I’ve found challenging, but I love the detective work involved in working out how to maintain and extend our builds.
• In general, building confidence as a developer. This is an ongoing process, and I’m not there yet, but recently I’ve started to notice a real shift in (a somewhat unconscious) mindset from “this is really challenging, how will I ever get this” to “I’m sure I’ll figure it out somehow”.
• Honestly, working with such a badass group of smart, hard-working, considerate people. I regularly find myself observing how people handle certain situations or approaches they take to solving problems and trying to mirror that behaviour.

(Note that around July 2021, Ursa Computing (i.e. where I originally started my apprenticeship) became part of a new company, Voltron Data, but for the sake of simplicity here, I refer to where I work as Voltron Data, even when talking about things that happened prior to July 2021.)

What good looks like in R

I asked people on Twitter for their opinions of good R package documentation in general, and Jonathan Sinclair highlighted the ‘examples’ section from dplyr::case_when, the beginning of which is shown below.

I think Jonathan is spot on in his assessment. To paraphrase, the highlights for him are:

• there is next to no prose or intro
• there are plenty of comments, as needed, to explain the examples
• there is a variety of different examples
• there are examples of what not to do.

This kind of documentation appeals to my skim-reading self. If I’m trying to accomplish a task, sometimes I just want to run some code and see what happens to get an intuitive feel for what a function does. While I am fully prepared to slow down and read the rest of the documentation, a “quick win” motivates me to invest the additional effort. It tells me that the developers of this code have prioritised making things easy to understand and that the time I am investing will pay off.

I’ve been skimming through the documentation of some tidyverse and related packages - as I consider many of these to be well documented and easy to read. Here are some things I’ve observed which I think one can do to make function examples look great:

• include the most basic usage of a function
• use very simple toy datasets or standard in-built datasets
• demonstrate non-obvious behaviours of a function
• demonstrate different parameter values/combinations where relevant
• demonstrate any unusual parameters
• demonstrate on different objects if appropriate
• sometimes go beyond the use of an individual function to include common mini-workflows
• group documentation and examples for similar functions together
• include examples that may lead to unexpected results
• include comments to explain examples
• no examples for deprecated functions to discourage their use
• no unpredictable external dependencies - rvest::html_text manually creates HTML to demonstrate capabilities rather than scraping an external site
• sometimes showing the output when it adds to the example (e.g. tidyselect::starts_with() and many other examples from that package)
• examples should be correct and run without error (unless intended to show erroneous output)

What bad looks like in R

I am not intending to “name and shame” any package authors who haven’t included examples for their functions. It may have been overlooked, there may be plenty of explanation elsewhere, or they may have felt that the code was not sufficiently complex to require examples. It might be true that it seems obvious what a function does, but that makes assumptions about the users of your code that might not hold.

What good looks like generally

When reading through examples, one thing that struck me is that when I’m looking at Python docs in Jupyter Notebook (press shift + tab), I also see the output of running the examples.

Similarly, both examples and outputs are shown in the official docs for some libraries, for example, pandas.

I think this is a helpful feature - less effort is required to see how a function works.

In R function documentation, runnable code is often included, but in most cases needs to be manually run by the reader to see the output. I’m torn as to whether this is good or not. On the one hand, it encourages you to run the code and get a more tangible feel for what it does and saves valuable space in the Viewer window in RStudio. On the other hand, it adds an extra manual step to your workflow and lengthens the time until that precious “quick win” of enlightenment when exploring a new function.

You get a lot closer to this on the website rdrr.io, which indexes R package documentation and allows examples to be run inline. However, examples are run one after the other without the original code being displayed. So in the case of multiple examples, you have to match up the output to which example it is from.

Some packages include output as comments within their examples. For instance, the tidyselect package; here’s an example from tidyselect::all_of:

All that said, while the ability to see the output of examples is a nice-to-have, I don’t think it’s essential to good function documentation. With any piece of documentation, it’s necessary to consider the purpose; at a minimum, examples exist to tell the reader how to use a function, and you don’t need to see the output to do that.

Since I first wrote this, I found out that it is possible to easily run examples from help files by selecting them and then hitting Ctrl+Enter, the same as running code in the Source pane.

Another thing I wasn’t aware of - pkgdown - commonly used to automatically render docs for packages run examples and displays the output underneath. Check out the example below from the Arrow pkgdown site.

In conclusion, good examples make functions easier to work with and help readers of your documentation gain a deeper understanding of how a function works. While any examples are better than no examples, you can give your users the best chance of success when using your code with careful thought about the content of your documentation.

Huge thanks to everyone who responded to my Twitter thread, and to my fantastic colleague Joris Van den Bossche for reading the first draft of this, and our conversations about how things are done in R and Python.

What good looks like in R

I asked people on Twitter for their opinions of good R package documentation in general, and Jonathan Sinclair highlighted the ‘examples’ section from dplyr::case_when, the beginning of which is shown below.

I think Jonathan is spot on in his assessment. To paraphrase, the highlights for him are:

• there is next to no prose or intro
• there are plenty of comments, as needed, to explain the examples
• there is a variety of different examples
• there are examples of what not to do.

This kind of documentation appeals to my skim-reading self. If I’m trying to accomplish a task, sometimes I just want to run some code and see what happens to get an intuitive feel for what a function does. While I am fully prepared to slow down and read the rest of the documentation, a “quick win” motivates me to invest the additional effort. It tells me that the developers of this code have prioritised making things easy to understand and that the time I am investing will pay off.

I’ve been skimming through the documentation of some tidyverse and related packages - as I consider many of these to be well documented and easy to read. Here are some things I’ve observed which I think one can do to make function examples look great:

• include the most basic usage of a function
• use very simple toy datasets or standard in-built datasets
• demonstrate non-obvious behaviours of a function
• demonstrate different parameter values/combinations where relevant
• demonstrate any unusual parameters
• demonstrate on different objects if appropriate
• sometimes go beyond the use of an individual function to include common mini-workflows
• group documentation and examples for similar functions together
• include examples that may lead to unexpected results
• include comments to explain examples
• no examples for deprecated functions to discourage their use
• no unpredictable external dependencies - rvest::html_text manually creates HTML to demonstrate capabilities rather than scraping an external site
• sometimes showing the output when it adds to the example (e.g. tidyselect::starts_with() and many other examples from that package)
• examples should be correct and run without error (unless intended to show erroneous output)

What bad looks like in R

I am not intending to “name and shame” any package authors who haven’t included examples for their functions. It may have been overlooked, there may be plenty of explanation elsewhere, or they may have felt that the code was not sufficiently complex to require examples. It might be true that it seems obvious what a function does, but that makes assumptions about the users of your code that might not hold.

What good looks like generally

When reading through examples, one thing that struck me is that when I’m looking at Python docs in Jupyter Notebook (press shift + tab), I also see the output of running the examples.

Similarly, both examples and outputs are shown in the official docs for some libraries, for example, pandas.

I think this is a helpful feature - less effort is required to see how a function works.

In R function documentation, runnable code is often included, but in most cases needs to be manually run by the reader to see the output. I’m torn as to whether this is good or not. On the one hand, it encourages you to run the code and get a more tangible feel for what it does and saves valuable space in the Viewer window in RStudio. On the other hand, it adds an extra manual step to your workflow and lengthens the time until that precious “quick win” of enlightenment when exploring a new function.

You get a lot closer to this on the website rdrr.io, which indexes R package documentation and allows examples to be run inline. However, examples are run one after the other without the original code being displayed. So in the case of multiple examples, you have to match up the output to which example it is from.

Some packages include output as comments within their examples. For instance, the tidyselect package; here’s an example from tidyselect::all_of:

All that said, while the ability to see the output of examples is a nice-to-have, I don’t think it’s essential to good function documentation. With any piece of documentation, it’s necessary to consider the purpose; at a minimum, examples exist to tell the reader how to use a function, and you don’t need to see the output to do that.

Since I first wrote this, I found out that it is possible to easily run examples from help files by selecting them and then hitting Ctrl+Enter, the same as running code in the Source pane.

Another thing I wasn’t aware of - pkgdown - commonly used to automatically render docs for packages run examples and displays the output underneath. Check out the example below from the Arrow pkgdown site.

In conclusion, good examples make functions easier to work with and help readers of your documentation gain a deeper understanding of how a function works. While any examples are better than no examples, you can give your users the best chance of success when using your code with careful thought about the content of your documentation.

Huge thanks to everyone who responded to my Twitter thread, and to my fantastic colleague Joris Van den Bossche for reading the first draft of this, and our conversations about how things are done in R and Python.