Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explicit headings in CSV export #23140

Merged
merged 9 commits into from
Aug 6, 2024

Conversation

jaads
Copy link
Member

@jaads jaads commented Jul 31, 2024

Scope

Currently when exporting a collection to CSV, the headings of the file were deducted from the first batch of the query result. In case the user choose to include relational data in the export and the first batch of items did not include any relational data, the headings did not exist and therefore the json2scv library put the values in the wrong 'column'.

In this PR, the field names which should be exported and hence the headings of the csv file, are determined beforehand instead of deducting those from the first batch.

An example

  • setup 13 articles, first 10 don't have an author related to it
  • set EXPORT_BATCH_SIZE of 10
  • specify those fields: id, title, author, author.first_name

Before

For the first batch, the author is always null. When deducting the headings this way, no first_name column is present in the export. For the second batch though, this value existed in the database response and was added in the CSV file in the wrong place. within the author column there should be the ID, and "Jan" should be in a column which doesn't exit.

title sort author
sgdfg 1
efa 2
sccx 3
sdfgdf 4
sdf 5
dsgfdghf 6
fgh 7
drg 8
d 9
hk 10
dg 11 Jan
fyjk 12 Jan
as 13 Jan

After

With the fields specified explicitly, the columns of the export did not rely on the first batch result anymore. This way the shifting could be eliminated.

id title author.first_name
5 sgdfg
6 efa
9 sccx
8 sdfgdf
4 sdf
10 dsgfdghf
7 fgh
14 drg
15 d
11 hk
12 dg Jan
13 fyjk Jan
3 as Jan

NOTE: I could not test it with the dataset provided in the linked issue, since this is a dump from v10.


Fixes #22298

Copy link

changeset-bot bot commented Jul 31, 2024

🦋 Changeset detected

Latest commit: 22f71f5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@directus/api Minor
directus Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@jaads jaads changed the title explicit fields for csv parser explicit fields for csv export Aug 2, 2024
@jaads jaads changed the title explicit fields for csv export Explicit headings in CSV export Aug 2, 2024
@jaads jaads marked this pull request as ready for review August 2, 2024 13:13
@jaads jaads self-assigned this Aug 2, 2024
@jaads jaads added this to the Next Major Release milestone Aug 2, 2024
Copy link
Member

@joselcvarela joselcvarela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and tested and seems to working as expected ✅

@jaads jaads merged commit 1c6481a into v11-rc Aug 6, 2024
4 checks passed
@jaads jaads deleted the fix/shifts-in-csv-export-for-small-batch-sizes branch August 6, 2024 08:30
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 6, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

API: Columns shift when EXPORT_BATCH_SIZE is surpassed on exporting CSV
4 participants