Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exif date/time isn't parsed correctly #9331

Closed
2 of 3 tasks
ygaeon opened this issue May 8, 2024 · 27 comments
Closed
2 of 3 tasks

Exif date/time isn't parsed correctly #9331

ygaeon opened this issue May 8, 2024 · 27 comments

Comments

@ygaeon
Copy link

ygaeon commented May 8, 2024

The bug

(Note this might be related to / same as #4898)

We have two different models of phones,

  • Fairphone 4, CalyxOS, OpenCamera v1.52 (mine)
  • Samsung Galaxy Note 10, Samsung Android (?), Samsung Camera v? (wifey's)

Both cameras sync to Immich server but our photos are not in order and I started investigating.

My photos are saved with wrong date/timestamp.
I'm currently in UTC+2 (Stockholm/Europe).
Comparing EXIFs (exiftool 12.76):

Photos

Fairphone 4:

File Name                       : IMG_20240504_164142.jpg

File Modification Date/Time     : 2024:05:04 16:41:42+02:00
File Access Date/Time           : 2024:05:06 19:05:53+02:00 *
File Inode Change Date/Time     : 2024:05:06 19:06:02+02:00 *

Date/Time Original              : 2024:05:04 16:41:42
Create Date                     : 2024:05:04 16:41:42

Create Date                     : 2024:05:04 16:41:42.809319
Date/Time Original              : 2024:05:04 16:41:42.809319
Modify Date                     : 2024:05:04 16:41:42.809319

* <- when I syncthing'd it

This Date/Time is translated to

05/04/2024, 06:41 PM (18:41)
Africa/Blantyre (+02:00)

Wrong time, wrong timezone ...

Samsung Galaxy Note 10:

File Name                       : 20240504_153307.jpg

File Modification Date/Time     : 2024:05:04 15:33:09+02:00
File Access Date/Time           : 2024:05:04 15:33:09+02:00
File Inode Change Date/Time     : 2024:05:06 20:02:35+02:00

Date/Time Original              : 2024:05:04 15:33:08
Create Date                     : 2024:05:04 15:33:08

Create Date <-- SGN10 doesn't have this at this pos
Date/Time Original              : 2024:05:04 15:33:08+02:00
Modify Date                     : 2024:05:04 15:33:08+02:00

This Date/Time is translated to

05/04/2024, 03:33 PM (15:33)
Africa/Blantyre (+02:00)

Correct time, wrong timezone ...

Videos

Fairphone 4:

File Name                       : VID_20240504_164249.mp4

File Modification Date/Time     : 2024:05:04 16:44:35+02:00
File Access Date/Time           : 2024:05:06 19:06:24+02:00 *
File Inode Change Date/Time     : 2024:05:06 19:06:34+02:00 *

Create Date                     : 2024:05:04 14:44:35
Modify Date                     : 2024:05:04 14:44:35

Track Create Date               : 2024:05:04 14:44:35
Track Modify Date               : 2024:05:04 14:44:35

Media Create Date               : 2024:05:04 14:44:35
Media Modify Date               : 2024:05:04 14:44:35

* <- when I syncthing'd it

This Date/Time is translated to

05/04/2024, 02:44 PM (14:44)
Africa/Abidjan (+00:00)

Which kind of translates to "correct" even if the correct should be "05/04/2024, 04:44 PM (16:44), Europe/Stockholm (+02:00)"

Samsung Galaxy Note 10:

File Name                       : 20240504_154055.mp4

File Modification Date/Time     : 2024:05:04 15:41:04+02:00
File Access Date/Time           : 2024:05:04 15:41:04+02:00
File Inode Change Date/Time     : 2024:05:06 20:02:52+02:00

Create Date                     : 2024:05:04 13:41:04
Modify Date                     : 2024:05:04 13:41:04

Track Create Date               : 2024:05:04 13:41:04
Track Modify Date               : 2024:05:04 13:41:04

Media Create Date               : 2024:05:04 13:41:04
Media Modify Date               : 2024:05:04 13:41:04

This Date/Time is translated to

05/04/2024, 03:41 PM (15:41)
Africa/Blantyre (+02:00)

Correct time, wrong timezone ...

From https://web.archive.org/web/20180921145139if_/http://www.cipa.jp:80/std/documents/e/DC-010-2017_E.pdf one can read:

  • DateTimeOriginal, DateTimeDigitized - The date and time when the image was stored as digital data (no time zone in Exif), stored in ISO 8601 format, not in the original Exif format. This property includes the value for the Exif SubSecTimeDigitized attribute.

From https://en.wikipedia.org/wiki/ISO_8601 on can read:

  • Time zones in ISO 8601 are represented as local time (with the location unspecified), as UTC, or as an offset from UTC. Time zone designators: <time>Z <time>±hh:mm <time>±hhmm <time>±hh
  • If no UTC relation information is given with a time representation, the time is assumed to be in local time.

Now, this is clearly not what's happening in Fairphone 4 case above:

  • "2024:05:04 16:41:42.809319" is being translated into "05/04/2024, 06:41 PM (18:41) Africa/Blantyre (+02:00)".
  • The date/time is already in UTC+2 ("local time").

Something is clearly not being parsed correctly somewhere .. and I have thousands of photos that are sorted wrongly in the archive now ..

Please advice ..

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.99.0

Version of Immich Mobile App

1.100.0 build.130

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

N/A .. running podman


podman pod create --name pod-immich --hostname immich --publish 2283:3001

podman run -d --pod pod-immich --name immich_redis redis:6.2-alpine

podman run -d --pod pod-immich --name immich_postgres --env-file $DP/immich.env --volume $DP/postgres:/var/lib/postgresql/data tensorchord/pgvecto-rs:pg14-v0.2.0

podman run -d --pod pod-immich --name immich_machine_learning --env-file $DP/immich.env --volume $DP/model-cache:/cache immich-machine-learning:v1.99.0

podman run -d --pod pod-immich --name immich_server --env-file $DP/immich.env --volume $DP/upload:/usr/src/app/upload --volume /etc/localtime:/etc/localtime:ro immich-server:v1.99.0 start.sh immich

podman run -d --pod pod-immich --name immich_microservices --env-file $DP/immich.env --volume $DP/upload:/usr/src/app/upload --volume /etc/localtime:/etc/localtime:ro immich-server:v1.99.0 start.sh microservices

Your .env content

# General
TZ="Europe/Stockholm"

# Database
DB_HOSTNAME=immich_postgres
DB_USERNAME=immich
DB_PASSWORD=xxxxx

DB_DATABASE=immich
DB_DATABASE_NAME=immich

# Postgres
POSTGRES_USER=immich
POSTGRES_PASSWORD=xxxxx
POSTGRES_DB=immich

# Redis
REDIS_HOSTNAME=immich_redis

Reproduction steps

N/A

Relevant log output

No response

Additional information

No response

@scrampker
Copy link

I'm seeing similar problems on my GrapheneOS phone as well. Seems to impact videos and photos differently. Sometimes tagged with local time zone, other times, UTC. -- but odd, because it's not just the wrong time zone, it can be the wrong day in UTC. IE videos I took last night are saying they're in the future -- for tonight.

@fmartingr
Copy link

I see similar issues on my end. In my case I my pictures where saved into iCloud Photos from a DJI Osmo 3, and when the app imported them into Immich the datetime was wrong by 2 hours.

I removed all pictures and tried to import them manually via
the web interface, same result.

Attaching one of the pictures that has that issue.

DJI_20240511164405_0094_D.JPG

@scrampker
Copy link

scrampker commented May 12, 2024 via email

@ygaeon
Copy link
Author

ygaeon commented May 20, 2024

Anyone from the project that can comment back?

@TekkertheChaot
Copy link

TekkertheChaot commented May 26, 2024

Chiming in; same behavior here with older photos shot on a Motorola Moto G (1. Gen), additionaly it dates it far back (photo from 2016, Immich dates it as 2002)
For comparison, Windows Explorer seems to parse the information correctly.
I have my external library bound via a read only share, might this be a cause for the metadata not being processed correctly? (I am aware that manual changes to the metadata to these files cannot be saved, as it has no write access on the source folder)

grafik

EDIT: Okay, I just learned that Windows Explorer doesn't always read EXIF data.
In my case, windows explorer read the base file attribute but there ACTUALLY IS a "Created"-Date with an actual 2002-date for some reason (inspected with ExifTool). Hopefully that helps some people further diagnosing their metadata ✌️
grafik

@ygaeon
Copy link
Author

ygaeon commented Jun 4, 2024

2 weeks later: "Anyone from the project that can comment back?"

@alextran1502
Copy link
Contributor

Are you on the latest version of the mobile app/server?

@ygaeon
Copy link
Author

ygaeon commented Jun 4, 2024

Was at the time of reporting, yes ..
.. and it's unfortunately a bit of a hassle to update so I refrained from doing so until had some feedback in this thread :-)

@alextran1502
Copy link
Contributor

I am pretty sure the latest version was v1.105.0 two weeks ago. Can you help upgrade it and test it again? Also include the troubled file will help with troubleshooting as well.

@fmartingr
Copy link

I just updated to v1.105.1 and refreshed the metadata for my entire library, still having the same issue. Sample file in my previous comment.

@ygaeon
Copy link
Author

ygaeon commented Jun 4, 2024

Just updated to

  • App version 1.105.1 build.140 and
  • Server version 1.105.1

Tested and it is still borked ..

@ygaeon
Copy link
Author

ygaeon commented Jun 6, 2024

Attached sample file ...

IMG_20240605_002021.jpg

That becomes like follows in Immich ...

2024-06-06_09-06

@ygaeon
Copy link
Author

ygaeon commented Jul 22, 2024

Almost 4 weeks later: "Anyone from the project that can comment back?"

@alextran1502
Copy link
Contributor

@ygaeon The timezone is parsed based on a few heuristics, so it isn't easy to explain precisely which one is used. From my observation in the past, CalyxOS strips GPS data, which the parsing mechanism uses to identify the correct timezone

You can read more here https://photostructure.github.io/exiftool-vendored.js/#md:dates

@ygaeon
Copy link
Author

ygaeon commented Aug 12, 2024

Thanks @alextran1502 and yes, CalyxOS do but I'm using "OpenCamera" which does not strip GPS, afaik. Though, I fail to see why GPS would be needed in this case as the exif data is present. Please just use it when it is there?

$ identify -verbose IMG_20240605_002021.jpg 
Image:
  Filename: IMG_20240605_002021.jpg
...
    exif:DateTime: 2024:06:05 00:20:20
    exif:DateTimeDigitized: 2024:06:05 00:20:20
    exif:DateTimeOriginal: 2024:06:05 00:20:20
...

@ygaeon
Copy link
Author

ygaeon commented Aug 26, 2024

I checked the attached file, following your link @alextran1502 ...

Heuristic 1: explicit metadata

  • My image doesn't have TimeZoneOffset
  • exif:DateTimeOriginal: 2024:06:05 00:20:20 is set
  • ModifyDate is NOT set
  • None of OffsetTime, OffsetTimeOriginal, nor OffsetTimeDigitized is set.

Heuristic 2: GPS location

No GPS location available

Heuristic 3: UTC timestamps

  • Neither GPSDateTime nor DateTimeUTC is set.

.. so I guess that exiftool-vendored would assume UTC in this case .. making all photos off by the 1-2 hours I see ..

In this case, would it not be possible to "attach" the timezone (from the mobile device) to the image/video being uploaded to the store?

@ygaeon
Copy link
Author

ygaeon commented Sep 13, 2024

Opened this over 4 months ago, still have the same issues. Anyone more knowledgeable have any comments or feedback?

Server Version: v1.114.0
App Version: 1.114.0 build.158

@radh21301
Copy link

radh21301 commented Sep 15, 2024

Some observations

Starting info:

  • Our image has dateTimeOriginal
  • Does not have GPS or any exif tags that represent the timezone

The way I observe immich working when it does not have any information related to timezone:

  • Assume that the time (Date Time Original or whatever it has) is in UTC
  • But uses the timezone of the server (set by an environment variable in the docker compose). Hence you see that GMT + 2:00. If the time in UTC was 00:20, then in the timezone the image was taken in, the local time was 02:20. Note: The browser and the app always displays the local time of wherever the image was taken. Hence, in the browser you see 02:20 GMT +2:00.

What would likely be an expected behavior is this:

  1. Assume a timezone for that image
  2. Then assume that the time (dateTimeOriginal or whatever the image has) is also in the same timezone it assumed in step 1
  • If you assumed timezone as UTC in step 1 --> Also assume that the dateTimeOriginal is in UTC
  • If you assumed timezone is the timezone of the server --> Also assume the dateTimeOriginal in server timezone. This is the better option in my opinion.

@Sean-T-Moore
Copy link

Many cameras without GPS just store DateTimeOriginal tags without timezone information. We often take pictures while on vacation in timezones that are not the same as the immich server and not in UTC. I don't think immich should ever assume what the timezone is. If it can't be determined from the image metadata, then the timezone should be explicitly stated by the user during import. This is only necessary because immich likes to attach a timezone to timestamps it stores. Other photo management systems do not care about timezone and the assumption is that the DateTimeOriginal is simply in whatever timezone where the photo was captured.

@radh21301
Copy link

radh21301 commented Sep 15, 2024

Correction to my previous comment - by timezone of the server, I actually meant the timezone the user sets using a env variable in the compose.

Also, I dont think that there is a solution without having a timezone attached to an image. I mean the entire concept of time is relative. So how would any application be able to provide a timeline view of images that can be taken in multiple timezones. Everything needs to be brought to a common reference, which is UTC, and we need the timezone to do it.

@Sean-T-Moore
Copy link

I agree it is nice to have the proper timezone attached to an image, but it was not deemed necessary when EXIF metadata was invented for use with digital cameras. I've been using the photo management site flickr for 20 years with images that don't have timezone information and the timeline features work just fine when you understand the time displayed is wherever the photo was captured. Flickr and other photo sites do not assume a timezone. Only the photographer knows the correct timezone in these cases so we need a way to pass that to immich during photo import if immich insists on storing timezone. If the photographer travels often, it can't be a fixed offset set from an env variable.

@radh21301
Copy link

radh21301 commented Sep 15, 2024

That is confusing. Maybe getting a bit off topic here - I still do not think flicker's timeline would work well ( it can still work. Bit is the timeline accurate - dont think so). What happen's if you upload 2 images.
A - 10 AM (no timezone information in the image. But it was actually taken in PST)
B - 11 AM EST.

Which image would come first in the timeline in Flicker? Flicker would need to assume a timezone for the images that dont have timezone OR just have B (11 am) come after A (10 am) just because of the time, which could most certainly be wrong.

Coming back to the original problem :), Immich does have environment variable that set the timezone of images that do not have a timezone. but it might also be worthwhile to be able to edit the timezone in the GUI i think.

@Sean-T-Moore
Copy link

It is a known drawback of EXIF that until recently it did not provide a way for cameras to record time-zone information. Any long-lived photo management system has had to deal with this. Flickr and others simply do not use timezone information for the timeline since historically it has not been available. Digital cameras existed for 20 years before geotagging became common and before the EXIF spec was updated to allow timezone with OffsetTimeOriginal. If you upload two photos to flickr, one with timezone data and one without, it will ignore the timezone and make no assumptions about timezone for the other. The timeline will still be correct and ordered exactly as captured. It would only be a problem if your timeline somehow contains photos captured by multiple photographers shooting simultaneously in different timezones.

@ygaeon
Copy link
Author

ygaeon commented Sep 15, 2024

Thanks for some activity in this issue .. :-)

@Sean-T-Moore : If it can't be determined from the image metadata, then the timezone should be explicitly stated by the user during import.

Well, if the timezone is missing in the metadata I think the closest truth would be the date and time and timezone set on the mobile device .. I mean, we usually update the time on our mobile device when we travel, right?

@Sean-T-Moore
Copy link

@ygaeon: I think the closest truth would be the date and time and timezone set on the mobile device

I import photos when I get home, not when I'm away in a different time zone. So mobile device timezone can't be used to set image metadata during import. Mobile devices really should be smart enough to set image metadata correctly already during image capture.

@fmartingr
Copy link

Maybe a section in the UI showing the latest import along files already stored with the option to modify them in batch, to check if everything is set ok or modify datetime settings accordingly.

@danieldietzler
Copy link
Member

Issues with parsing date/time incorrectly should go to #12650, potential feature requests (bulk modify assets) should be a feature request (pretty sure there already is one anyways)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants