Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected app crush (502 response code) #12809

Closed
1 of 3 tasks
illmouse opened this issue Sep 20, 2024 · 2 comments
Closed
1 of 3 tasks

Unexpected app crush (502 response code) #12809

illmouse opened this issue Sep 20, 2024 · 2 comments

Comments

@illmouse
Copy link

The bug

Apparently during some maintenance (because it happened at 00:00), app became unavailable. Other containers, according to logs, is still app and running.

This issue didn't drop main PID in container so container hasn't been restarted properly.

The OS that Immich Server is running on

ghcr.io/immich-app/immich-server:release

Version of Immich Server

1.115.0

Version of Immich Mobile App

1.115.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    networks:
      - immich-frontend
      - immich-backend
    env_file:
      - .env
    #ports:
    #  - 2283:3001
    #expose:
    #  - "3001"
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    deploy:
      resources:
        limits:
          cpus: '0.5'
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    networks:
      - immich-backend
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b
    healthcheck:
      test: redis-cli ping || exit 1
    volumes:
      - redis-data:/data
    networks:
      - immich-backend
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    networks:
      - immich-backend
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_>
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "ma>
    restart: always

volumes:
  model-cache:
  redis-data:

networks:
  immich-frontend:
    name: immich-frontend
    external: true
  immich-backend:
    name: immich-backend
    external: true

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/immich-data/
# The location where your database files are stored
DB_DATA_LOCATION=/mnt/immich-db/

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zon>
# TZ=Etc/UTC
TZ=Europe/Moscow

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=xxx

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=xxx
DB_DATABASE_NAME=immich

IMMICH_ENV=production
CPU_CORES=4

Reproduction steps

Not sure, app became unavailable at 00:00. Perhaps during scheduled maintenance.

Relevant log output

[Nest] 6  - 09/20/2024, 12:00:00 AM   ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-preview): Error: VipsJpeg: Corrupt JPEG data: bad Huffman code
VipsJpeg: Corrupt JPEG data: bad Huffman code
VipsJpeg: Premature end of input file
[Nest] 6  - 09/20/2024, 12:00:00 AM   ERROR [Microservices:JobService] Error: VipsJpeg: Corrupt JPEG data: bad Huffman code
VipsJpeg: Corrupt JPEG data: bad Huffman code
VipsJpeg: Premature end of input file
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:90:19)
    at MediaRepository.generateThumbnail (/usr/src/app/dist/repositories/media.repository.js:69:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:176:48)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:141:29)
    at async /usr/src/app/dist/services/job.service.js:148:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 6  - 09/20/2024, 12:00:00 AM   ERROR [Microservices:JobService] Object:
{
  "id": "dc4f79d6-e58f-4444-a080-fee8cb5e5f3a"
}

Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 09/20/2024, 12:00:49 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 15  - 09/20/2024, 12:00:49 AM     LOG [Api:EventRepository] Initialized websocket server

Additional information

No response

@illmouse
Copy link
Author

Sorry, further actions to get app back up showed there is a problem with postgres container. I will further investigate the cause.

@illmouse
Copy link
Author

Looks like it was a problem that mounted LUN storing DB became unresponsive. So no real problem with immich.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant