text:
NFS  Issues / worker stops after one file, and does not restart

Honestly, I am not that great in explaining what actually is happening.
I quickly set up a debian VM,  installed docker, nfstools and so on.
Mounted my NFS drive (v4.1, async etc), volume mounted these drives to librephotos.
Setup libre photos, start a scan which stops or hangs after one photo. I stop the scanning process and restart it -> another photo is added to the library, and it hangs again. I can repeat this process until an entire folder is scanned but that's probably not wanted.

Are there error messages in the log files? Does it work if you scan a folder on your local drive? Which information does the photo have when you click on it? Is it geocoded, has it scene attributes or similar images?

Okidoki:
So first things first
d-c.yaml
# DO NOT EDIT
# The .env file has everything you need to edit.
# Run options:
# 1. Use prebuilt images (preferred method):
#   run cmd: docker-compose up -d
# 2. Build images on your own machine:
#   build cmd: COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build
#   run cmd: docker-compose up -d

version: '2.4'
services:

  proxy:
    image: reallibrephotos/librephotos-proxy:${tag}
    tty: true
    container_name: librephotos-proxy
    restart: always
    ports:
      - ${httpPort}:80

  librephotos-db:
    image: postgres
    container_name: librephotos-db
    restart: always
    environment:
      - POSTGRES_USER=${dbUser}
      - POSTGRES_PASSWORD=${dbPass}
      - POSTGRES_DB=${dbName}
    volumes:
      - ${dbLocation}:/var/lib/postgresql/data
    command: postgres -c fsync=off -c synchronous_commit=off -c full_page_writes=off -c random_page_cost=1.0
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $dbName -U $dbUser"]
      interval: 5s
      timeout: 5s
      retries: 5

  frontend:
    image: reallibrephotos/librephotos-frontend:${tag}
    container_name: librephotos-frontend
    restart: always
    tty: true

  backend:
    image: reallibrephotos/librephotos:${tag}
    container_name: librephotos-backend
    restart: always
    volumes:
      - ${myPhotos}:/data
      #- /home/docker/photos:/data
      - ${proMedia}:/code/protected_media
      - ${logLocation}:/code/logs
      - ${cachedir}:/root/.cache

    environment:
      - SECRET_KEY=${shhhhKey}
      - BACKEND_HOST=backend
      - ADMIN_EMAIL=${adminEmail}
      - ADMIN_USERNAME=${userName}
      - ADMIN_PASSWORD=${userPass}
      - DEBUG=true
      - DB_BACKEND=postgresql
      - DB_NAME=${dbName}
      - DB_USER=${dbUser}
      - DB_PASS=${dbPass}
      - DB_HOST=librephotos-db
      - DB_PORT=5432
      - REDIS_HOST=librephotos-redis
      - REDIS_PORT=6379
      - MAPBOX_API_KEY=${mapApiKey}
      - TIME_ZONE=${timeZone}
      - WEB_CONCURRENCY=${gunniWorkers}
      - WORKER_TIMEOUT=${workerTimeOut}
    # Wait for Postgres
    depends_on:
      librephotos-db:
        condition: service_healthy

  librephotos-redis:
    image: redis:alpine
    container_name: librephotos-redis
    restart: always

excerpts .env
myPhotos=/mnt/photos/   # NFS location
# Where shall we store the cache files.
cachedir=/home/docker/librephotos/librephotos_cache  #local
# Where shall we store the database.
dbLocation=/home/docker/librephotos/librephotos_data #local

Created a small library with some images (19 in total, it scanned in 2) some compact camera jpgs of my dog
File Path /data/fuji/2021_0206/DSCF0378.JPG

image no geotaggen as far as i can tell

Removed the task and retried the folder scan

no logs created. when i docker-compose up i get this

If you want me to remove the database and create a new one i can, let me know

I quickly recreate the docker. need a couple mins for that

logs and size

2021-02-11 15:56:00,701 : views.py : list : 841 : INFO : querying & serializing took 0.00 seconds
2021-02-11 15:56:12,957 : serializers.py : update : 705 : INFO : Updated scan directory for user /data/fuji/2021_0206
2021-02-11 15:56:12,957 : serializers.py : update : 711 : INFO : Updated confidence for user 0.1
2021-02-11 15:56:21,960 : directory_watcher.py : handle_new_image : 86 : INFO : job 7da97790-a9e5-4cf7-9d28-023841191a6d: handling image /data/fuji/2021_0206/DSCF0378.JPG
2021-02-11 15:56:23,623 : photo.py : _generate_captions : 125 : INFO : generated places365 captions for image /code/protected_media/thumbnails_big/3148a85ffb9832ed813ddef4566096161.jpg.
2021-02-11 15:56:38,318 : directory_watcher.py : handle_new_image : 120 : INFO : job 7da97790-a9e5-4cf7-9d28-023841191a6d: image processed: /data/fuji/2021_0206/DSCF0378.JPG, elapsed: 16.34067
2021-02-11 15:56:38,323 : directory_watcher.py : handle_new_image : 86 : INFO : job 7da97790-a9e5-4cf7-9d28-023841191a6d: handling image /data/fuji/2021_0206/DSCF0376.JPG
2021-02-11 15:56:39,786 : photo.py : _generate_captions : 125 : INFO : generated places365 captions for image /code/protected_media/thumbnails_big/7c60545386b046a0dae6b9e4a46601521.jpg.
2021-02-11 15:56:46,291 : photo.py : _extract_faces : 368 : INFO : image 7c60545386b046a0dae6b9e4a46601521: 1 face(s) saved
2021-02-11 15:56:46,706 : views.py : get : 1582 : ERROR : An Error occured
Traceback (most recent call last):
  File "/code/api/views.py", line 1579, in get
    scan_photos(request.user, job_id)
  File "/code/api/directory_watcher.py", line 234, in scan_photos
    walk_directory(user.scan_directory, ps)  # now walk with photo-scannning
  File "/code/api/directory_watcher.py", line 177, in walk_directory
    callback.read_file(fpath)
  File "/code/api/directory_watcher.py", line 206, in read_file
    handle_new_image(self.user, path, self.job_id)
  File "/code/api/directory_watcher.py", line 117, in handle_new_image
    photo._im2vec()
  File "/code/api/models/photo.py", line 320, in _im2vec
    im2vec = Im2Vec(cuda=False)
  File "/code/api/im2vec.py", line 20, in __init__
    self.model, self.extraction_layer = self._get_model_and_layer(model, layer)
  File "/code/api/im2vec.py", line 66, in _get_model_and_layer
    model = models.resnet152(pretrained=True)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 288, in resnet152
    return _resnet('resnet152', Bottleneck, [3, 8, 36, 3], pretrained, progress,
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 224, in _resnet
    model = ResNet(block, layers, **kwargs)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 162, in __init__
    nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
  File "/miniconda/lib/python3.8/site-packages/torch/nn/init.py", line 418, in kaiming_normal_
    return tensor.normal_(0, std)
  File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)
SystemExit: 1

Found a couple of issues and fixed them. I am not yet sure if it will fix your problem, but it should report now better error messages. The image is currently building. Please do a docker-compose down and a docker-compose pull and then a docker-compose up in half an hour :)

2021-02-11 19:39:50,566 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-11 19:39:57,199 : directory_watcher.py : handle_new_image : 86 : INFO : job 6a03eb03-ee98-4034-96c1-04c0a38d4697: handling image /data/fuji/2021_0206/DSCF0371.JPG
2021-02-11 19:39:58,790 : photo.py : _generate_captions : 125 : INFO : generated places365 captions for image /code/protected_media/thumbnails_big/4d7023a3607874191459322470df0e8e1.jpg.
2021-02-11 19:40:05,537 : photo.py : _extract_faces : 365 : INFO : image 4d7023a3607874191459322470df0e8e1: 1 face(s) saved
2021-02-11 19:40:07,185 : photo.py : _im2vec : 325 : ERROR : something went wrong with im2vec
Traceback (most recent call last):
  File "/code/api/models/photo.py", line 319, in _im2vec
    im2vec = Im2Vec(cuda=False)
  File "/code/api/im2vec.py", line 20, in __init__
    self.model, self.extraction_layer = self._get_model_and_layer(model, layer)
  File "/code/api/im2vec.py", line 66, in _get_model_and_layer
    model = models.resnet152(pretrained=True)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 288, in resnet152
    return _resnet('resnet152', Bottleneck, [3, 8, 36, 3], pretrained, progress,
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 226, in _resnet
    state_dict = load_state_dict_from_url(model_urls[arch],
  File "/miniconda/lib/python3.8/site-packages/torch/hub.py", line 559, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location)
  File "/miniconda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/miniconda/lib/python3.8/site-packages/torch/serialization.py", line 749, in _legacy_load
    return legacy_load(f)
  File "/miniconda/lib/python3.8/site-packages/torch/serialization.py", line 674, in legacy_load
    tar.extract('storages', path=tmpdir)
  File "/miniconda/lib/python3.8/tarfile.py", line 2067, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/miniconda/lib/python3.8/tarfile.py", line 2139, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/miniconda/lib/python3.8/tarfile.py", line 2188, in makefile
    copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
  File "/miniconda/lib/python3.8/tarfile.py", line 250, in copyfileobj
    dst.write(buf)
  File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)
SystemExit: 1
2021-02-11 19:40:07,189 : directory_watcher.py : handle_new_image : 120 : INFO : job 6a03eb03-ee98-4034-96c1-04c0a38d4697: image processed: /data/fuji/2021_0206/DSCF0371.JPG, elapsed: 9.905661
2021-02-11 19:40:07,244 : directory_watcher.py : handle_new_image : 86 : INFO : job 6a03eb03-ee98-4034-96c1-04c0a38d4697: handling image /data/fuji/2021_0206/DSCF0367.JPG
2021-02-11 19:40:22,971 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-11 19:40:33,304 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-11 19:40:34,938 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-11 19:42:02,680 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-11 19:42:04,574 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-11 19:42:07,986 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds

this time it indexed 2 images and then stalled again.

I hope that helps, if i need to get you more detail just tell me where to find it :D

Which CPU are you using? Somebody in #94 has the same issue.

@derneuere here's my /proc/cpuinfo:
https://pastebin.com/XaPsHUKa

i am using a AMD 3900X 12 cores for a debian vm

I think I found the issue, but I am not sure yet. The image is currently building. Please do a docker-compose down and a docker-compose pull and then a docker-compose up in half an hour :)

will do it now, sorry for the delay :)
updated, still stalls

021-02-13 09:09:49,699 : directory_watcher.py : handle_new_image : 120 : INFO : job abacc9ae-b6ae-4654-a3f2-1302a6c790b4: image processed: /data/fuji/2021_0206/DSCF0379.JPG, elapsed: 9.117626
2021-02-13 09:09:49,868 : directory_watcher.py : handle_new_image : 86 : INFO : job abacc9ae-b6ae-4654-a3f2-1302a6c790b4: handling image /data/fuji/2021_0206/DSCF0368.JPG
2021-02-16 05:58:07,409 : image_similarity.py : build_image_similarity_index : 33 : INFO : builing similarity index for user zwck-admin
2021-02-16 19:28:58,730 : image_similarity.py : build_image_similarity_index : 33 : INFO : builing similarity index for user zwck-admin
2021-02-16 19:29:27,755 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-16 19:29:53,287 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-16 19:29:59,343 : directory_watcher.py : handle_new_image : 86 : INFO : job 0c4b9a9a-cbef-4538-bafe-2d2d812d1b15: handling image /data/fuji/2021_0206/DSCF0366.JPG
2021-02-16 19:30:01,178 : photo.py : _generate_captions : 125 : INFO : generated places365 captions for image /code/protected_media/thumbnails_big/003c9ccf614b648ba813424c32d31fb01.jpg.
2021-02-16 19:30:08,209 : photo.py : _extract_faces : 365 : INFO : image 003c9ccf614b648ba813424c32d31fb01: 1 face(s) saved
2021-02-16 19:30:08,704 : photo.py : _im2vec : 325 : ERROR : something went wrong with im2vec
Traceback (most recent call last):
  File "/code/api/models/photo.py", line 319, in _im2vec
    im2vec = Im2Vec(cuda=False)
  File "/code/api/im2vec.py", line 20, in __init__
    self.model, self.extraction_layer = self._get_model_and_layer(model, layer)
  File "/code/api/im2vec.py", line 66, in _get_model_and_layer
    model = models.resnet152(pretrained=True)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 288, in resnet152
    return _resnet('resnet152', Bottleneck, [3, 8, 36, 3], pretrained, progress,
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 224, in _resnet
    model = ResNet(block, layers, **kwargs)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 162, in __init__
    nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
  File "/miniconda/lib/python3.8/site-packages/torch/nn/init.py", line 418, in kaiming_normal_
    return tensor.normal_(0, std)
  File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)
SystemExit: 1
2021-02-16 19:30:08,706 : directory_watcher.py : handle_new_image : 120 : INFO : job 0c4b9a9a-cbef-4538-bafe-2d2d812d1b15: image processed: /data/fuji/2021_0206/DSCF0366.JPG, elapsed: 9.272708
2021-02-16 19:30:08,725 : directory_watcher.py : handle_new_image : 86 : INFO : job 0c4b9a9a-cbef-4538-bafe-2d2d812d1b15: handling image /data/fuji/2021_0206/DSCF0381.JPG
2021-02-16 19:31:46,407 : directory_watcher.py : handle_new_image : 86 : INFO : job 88c3ce25-ec8b-412f-9a15-0c62581f3b75: handling image /data/fuji/2021_0206/DSCF0365.JPG
2021-02-16 19:31:48,094 : photo.py : _generate_captions : 125 : INFO : generated places365 captions for image /code/protected_media/thumbnails_big/f8f1e153e9091794747b25e0a79e1d1b1.jpg.
2021-02-16 19:31:55,944 : photo.py : _im2vec : 325 : ERROR : something went wrong with im2vec
Traceback (most recent call last):
  File "/code/api/models/photo.py", line 319, in _im2vec
    im2vec = Im2Vec(cuda=False)
  File "/code/api/im2vec.py", line 20, in __init__
    self.model, self.extraction_layer = self._get_model_and_layer(model, layer)
  File "/code/api/im2vec.py", line 66, in _get_model_and_layer
    model = models.resnet152(pretrained=True)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 288, in resnet152
    return _resnet('resnet152', Bottleneck, [3, 8, 36, 3], pretrained, progress,
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 224, in _resnet
    model = ResNet(block, layers, **kwargs)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 162, in __init__
    nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
  File "/miniconda/lib/python3.8/site-packages/torch/nn/init.py", line 418, in kaiming_normal_
    return tensor.normal_(0, std)
  File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)
SystemExit: 1
2021-02-16 19:31:55,950 : directory_watcher.py : handle_new_image : 120 : INFO : job 88c3ce25-ec8b-412f-9a15-0c62581f3b75: image processed: /data/fuji/2021_0206/DSCF0365.JPG, elapsed: 9.486586
2021-02-16 19:31:55,984 : directory_watcher.py : handle_new_image : 86 : INFO : job 88c3ce25-ec8b-412f-9a15-0c62581f3b75: handling image /data/fuji/2021_0206/DSCF0374.JPG

2021-02-16 19:31:55,984 : directory_watcher.py : handle_new_image : 86 : INFO : job 88c3ce25-ec8b-412f-9a15-0c62581f3b75: handling image /data/fuji/2021_0206/DSCF0374.JPG
2021-02-17 09:08:00,108 : image_similarity.py : build_image_similarity_index : 33 : INFO : builing similarity index for user zwck-admin
2021-02-17 09:08:16,831 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-17 09:08:32,764 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-17 09:08:37,897 : directory_watcher.py : handle_new_image : 86 : INFO : job 90a136d9-79b6-43cf-9b49-d6d5d93d8ca8: handling image /data/fuji/2021_0206/DSCF0373.JPG
2021-02-17 09:08:39,503 : photo.py : _generate_captions : 125 : INFO : generated places365 captions for image /code/protected_media/thumbnails_big/93c682c69028f026c15d8b9dfc1aede51.jpg.
2021-02-17 09:08:46,249 : photo.py : _extract_faces : 365 : INFO : image 93c682c69028f026c15d8b9dfc1aede51: 2 face(s) saved
2021-02-17 09:08:47,034 : photo.py : _im2vec : 325 : ERROR : something went wrong with im2vec
Traceback (most recent call last):
  File "/code/api/models/photo.py", line 319, in _im2vec
    im2vec = Im2Vec(cuda=False)
  File "/code/api/im2vec.py", line 20, in __init__
    self.model, self.extraction_layer = self._get_model_and_layer(model, layer)
  File "/code/api/im2vec.py", line 66, in _get_model_and_layer
    model = models.resnet152(pretrained=True)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 288, in resnet152
    return _resnet('resnet152', Bottleneck, [3, 8, 36, 3], pretrained, progress,
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 224, in _resnet
    model = ResNet(block, layers, **kwargs)
  File "/miniconda/lib/python3.8/site-packages/torchvision/models/resnet.py", line 162, in __init__
    nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
  File "/miniconda/lib/python3.8/site-packages/torch/nn/init.py", line 418, in kaiming_normal_
    return tensor.normal_(0, std)
  File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)
SystemExit: 1
2021-02-17 09:08:47,035 : directory_watcher.py : handle_new_image : 120 : INFO : job 90a136d9-79b6-43cf-9b49-d6d5d93d8ca8: image processed: /data/fuji/2021_0206/DSCF0373.JPG, elapsed: 9.034618
2021-02-17 09:08:47,144 : directory_watcher.py : handle_new_image : 86 : INFO : job 90a136d9-79b6-43cf-9b49-d6d5d93d8ca8: handling image /data/fuji/2021_0206/DSCF0380.JPG

Alright, I switched now to use a different model that is also used for places365. The places365 algorithm seems to work, so I hope that this will resolve your problem. The image is currently building. Please do a docker-compose down and a docker-compose pull and then a docker-compose up in half an hour :)

sorry to bring always bad news
2021-02-17 09:08:47,144 : directory_watcher.py : handle_new_image : 86 : INFO : job 90a136d9-79b6-43cf-9b49-d6d5d93d8ca8: handling image /data/fuji/2021_0206/DSCF0380.JPG
2021-02-17 15:25:20,265 : image_similarity.py : build_image_similarity_index : 33 : INFO : builing similarity index for user zwck-admin
2021-02-17 15:26:20,722 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-17 15:26:30,586 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-17 15:26:39,123 : directory_watcher.py : handle_new_image : 86 : INFO : job 9f7d1851-01da-45af-a8b6-a1e51030ff07: handling image /data/fuji/2021_0206/DSCF0372.JPG
2021-02-17 15:26:40,796 : photo.py : _generate_captions : 125 : INFO : generated places365 captions for image /code/protected_media/thumbnails_big/7ec541309a5b505010d9903db6998c831.jpg.
2021-02-17 15:26:47,634 : photo.py : _extract_faces : 365 : INFO : image 7ec541309a5b505010d9903db6998c831: 3 face(s) saved
2021-02-17 15:26:48,382 : photo.py : _im2vec : 325 : ERROR : something went wrong with im2vec
Traceback (most recent call last):
  File "/code/api/models/photo.py", line 319, in _im2vec
    im2vec = Im2Vec(cuda=False)
  File "/code/api/im2vec.py", line 20, in __init__
    self.model, self.extraction_layer = self._get_model_and_layer(model, layer)
  File "/code/api/im2vec.py", line 66, in _get_model_and_layer
    model = models.resnet152(pretrained=True)
  File "/code/wideresnet.py", line 209, in resnet152
    model = ResNet(Bottleneck, [3, 8, 36, 3], **kwargs)
  File "/code/wideresnet.py", line 115, in __init__
    m.weight.data.normal_(0, math.sqrt(2. / n))
  File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)
SystemExit: 1
2021-02-17 15:26:48,384 : directory_watcher.py : handle_new_image : 120 : INFO : job 9f7d1851-01da-45af-a8b6-a1e51030ff07: image processed: /data/fuji/2021_0206/DSCF0372.JPG, elapsed: 9.20524
2021-02-17 15:26:48,451 : directory_watcher.py : handle_new_image : 86 : INFO : job 9f7d1851-01da-45af-a8b6-a1e51030ff07: handling image /data/fuji/2021_0206/DSCF0375.JPG
2021-02-17 15:27:13,892 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds

could this be a problem on my end ? willing to try things

I have no idea, what I could try next tbh 😅 Maybe don't run it via a nfs mount, but directly in the vm with a couple of pictures. Or maybe try it on a regular linux machine.
Maybe it is hardware related. Do you have a gpu installed?

Hello, first of all I would like to thank you to take this project and revive back ownphoto project.
I am having a similar setup where I am running the latest docker images as of 4 hours ago. My setup is not using a NFS mount but a SAMBA share mount in a VM on ESXi. The VM OS is Ubuntu 18.04, Server CPU is Intel XEON E5-2630L v2, server model is HP DL360P G8 with 96GB RAM, VM has been allocated 12 cores with 16GB of RAM and 600GB of SSD space. Nothing else is running on this VM other than some docker applications. I have no GPU in this server. All the pictures are located in a WD NAS. Library is about 1.5TB, mixture of videos and JPEG with over 100k files. No 4K content most videos are shorter clips taken with cell phones.
I have Librephoto up for 10 days now and it has only scanned 1425 pictures. the scan actually stops after every 25 to 50 pictures.
adding the log below
Librephotos-backend  | Traceback (most recent call last): Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner Librephotos-backend  |     response = get_response(request) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response Librephotos-backend  |     response = self.process_exception_by_middleware(e, request) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response Librephotos-backend  |     response = wrapped_callback(request, *callback_args, **callback_kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view Librephotos-backend  |     return view_func(*args, **kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view Librephotos-backend  |     return self.dispatch(request, *args, **kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch Librephotos-backend  |     response = self.handle_exception(exc) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception Librephotos-backend  |     self.raise_uncaught_exception(exc) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Librephotos-backend  |     raise exc Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch Librephotos-backend  |     response = handler(request, *args, **kwargs) Librephotos-backend  |   File "/code/api/views.py", line 1424, in get Librephotos-backend  |     search_term_examples = get_search_term_examples(request.user) Librephotos-backend  |   File "/code/api/api_util.py", line 163, in get_search_term_examples Librephotos-backend  |     term_thing = random.choice(datum['things']) Librephotos-backend  |   File "/miniconda/lib/python3.8/random.py", line 290, in choice Librephotos-backend  |     raise IndexError('Cannot choose from an empty sequence') from None Librephotos-backend  | IndexError: Cannot choose from an empty sequence Librephotos-backend  | 20:13:53 Cleaning registries for queue: default Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | 20:40:53 Cleaning registries for queue: default Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Bad Request: /api/nextcloud/listdir/ Librephotos-backend  | Not Found: /api/jobs/ Librephotos-backend  | Internal Server Error: /api/searchtermexamples/ Librephotos-backend  | Traceback (most recent call last): Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner Librephotos-backend  |     response = get_response(request) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response Librephotos-backend  |     response = self.process_exception_by_middleware(e, request) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response Librephotos-backend  |     response = wrapped_callback(request, *callback_args, **callback_kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view Librephotos-backend  |     return view_func(*args, **kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view Librephotos-backend  |     return self.dispatch(request, *args, **kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch Librephotos-backend  |     response = self.handle_exception(exc) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception Librephotos-backend  |     self.raise_uncaught_exception(exc) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Librephotos-backend  |     raise exc Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch Librephotos-backend  |     response = handler(request, *args, **kwargs) Librephotos-backend  |   File "/code/api/views.py", line 1424, in get Librephotos-backend  |     search_term_examples = get_search_term_examples(request.user) Librephotos-backend  |   File "/code/api/api_util.py", line 163, in get_search_term_examples Librephotos-backend  |     term_thing = random.choice(datum['things']) Librephotos-backend  |   File "/miniconda/lib/python3.8/random.py", line 290, in choice Librephotos-backend  |     raise IndexError('Cannot choose from an empty sequence') from None Librephotos-backend  | IndexError: Cannot choose from an empty sequence Librephotos-backend  | Not Found: /api/jobs/ Librephotos-backend  | Not Found: /api/jobs/ Librephotos-backend  | Bad Request: /api/nextcloud/listdir/ Librephotos-backend  | Not Found: /api/jobs/ Librephotos-backend  | Bad Request: /api/nextcloud/listdir/ Librephotos-backend  | Not Found: /api/jobs/ librephotos-redis    | 1:M 17 Feb 2021 21:02:58.002 * 1 changes in 3600 seconds. Saving... librephotos-redis    | 1:M 17 Feb 2021 21:02:58.002 * Background saving started by pid 17 librephotos-redis    | 17:C 17 Feb 2021 21:02:58.006 * DB saved on disk librephotos-redis    | 17:C 17 Feb 2021 21:02:58.007 * RDB: 0 MB of memory used by copy-on-write librephotos-redis    | 1:M 17 Feb 2021 21:02:58.103 * Background saving terminated with success Librephotos-backend  | Internal Server Error: /api/searchtermexamples/ Librephotos-backend  | Traceback (most recent call last): Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner Librephotos-backend  |     response = get_response(request) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response Librephotos-backend  |     response = self.process_exception_by_middleware(e, request) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response Librephotos-backend  |     response = wrapped_callback(request, *callback_args, **callback_kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view Librephotos-backend  |     return view_func(*args, **kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view Librephotos-backend  |     return self.dispatch(request, *args, **kwargs) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch Librephotos-backend  |     response = self.handle_exception(exc) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception Librephotos-backend  |     self.raise_uncaught_exception(exc) Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Librephotos-backend  |     raise exc Librephotos-backend  |   File "/miniconda/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch Librephotos-backend  |     response = handler(request, *args, **kwargs) Librephotos-backend  |   File "/code/api/views.py", line 1424, in get Librephotos-backend  |     search_term_examples = get_search_term_examples(request.user) Librephotos-backend  |   File "/code/api/api_util.py", line 163, in get_search_term_examples Librephotos-backend  |     term_thing = random.choice(datum['things']) Librephotos-backend  |   File "/miniconda/lib/python3.8/random.py", line 290, in choice Librephotos-backend  |     raise IndexError('Cannot choose from an empty sequence') from None Librephotos-backend  | IndexError: Cannot choose from an empty sequence Librephotos-backend  | Bad Request: /api/nextcloud/listdir/ Librephotos-backend  | Not Found: /api/jobs/ Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | 21:07:53 Cleaning registries for queue: default Librephotos-backend  | Unauthorized: /api/rqavailable/ Librephotos-backend  | Unauthorized: /api/rqavailable/

@kaushikray1 This error is just the search trying to find examples, but getting something that is empty instead.  This should not be the reason why the scanning stops. I will look into it anyway.
Your error is probably a different error to what has been reported a couple posts above. Could you look if a gunicorn worker is timing out? Should look something like this: [CRITICAL] WORKER TIMEOUT (pid:150),

I have no idea, what I could try next tbh 😅 Maybe don't run it via a nfs mount, but directly in the vm with a couple of pictures. Or maybe try it on a regular linux machine.
Maybe it is hardware related. Do you have a gpu installed?

I see. There is not a gpu installed, i tried two different approaches, one with a DebianVM + NFS mount and one AlpineVM + NFS mounts (this is NFS 4.1). When I run it from the same hard drive, it works perfectly fine, I copied over 4GB of data and indexed it quite easily. But obviously that's not wanted haha :)
Not sure which hardware could be at fault here :) If you have an idea I'll try a different machine, I got pretty much everything imaginable. If its the AMD cpu i'll transfer the vm to an intel cpu.

This is driving me crazy :D Could you try this here: https://docs.photoprism.org/getting-started/faq/#how-can-i-mount-network-shares-with-docker
Photoprism also uses docker-compose and they recommend creating a volume.

of course

Done that, to no stops after 1 or two files.
volumes:
  volume_mounted_photos:
    driver: local
    driver_opts:
      type: nfs
      # The IP of your NAS:
      o: "addr=192.168.0.20,soft,rw"
      # Share path on your NAS:
      device: ":/mnt/private/data/sorted-data/photos"

  backend:
    image: reallibrephotos/librephotos:${tag}
    container_name: librephotos-backend
    security_opt:
      - seccomp:unconfined
      - apparmor:unconfined
    restart: always
    volumes:
      #- ${myPhotos}:/data
      - volume_mounted_photos:/data
      #- /home/docker/photos:/data

SystemExit: 1
2021-02-18 09:19:28,406 : serializers.py : update : 705 : INFO : Updated scan directory for user /data/Kai/Mauricio_Wedding
2021-02-18 09:19:28,406 : serializers.py : update : 711 : INFO : Updated confidence for user 0.1
2021-02-18 09:19:34,924 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds
2021-02-18 09:19:36,427 : directory_watcher.py : handle_new_image : 86 : INFO : job 7c92acf1-dea7-47c6-ae88-f76f7374d791: handling image /data/Kai/Mauricio_Wedding/my_favorits/IMG_5617.JPG
2021-02-18 09:19:37,871 : photo.py : _generate_captions : 125 : INFO : generated places365 captions for image /code/protected_media/thumbnails_big/bf54ca6118cb47412596ddd16438dde81.jpg.
2021-02-18 09:19:41,523 : photo.py : _extract_faces : 365 : INFO : image bf54ca6118cb47412596ddd16438dde81: 2 face(s) saved
2021-02-18 09:19:45,783 : directory_watcher.py : handle_new_image : 120 : INFO : job 7c92acf1-dea7-47c6-ae88-f76f7374d791: image processed: /data/Kai/Mauricio_Wedding/my_favorits/IMG_5617.JPG, elapsed: 9.32329
2021-02-18 09:19:45,928 : directory_watcher.py : handle_new_image : 86 : INFO : job 7c92acf1-dea7-47c6-ae88-f76f7374d791: handling image /data/Kai/Mauricio_Wedding/my_favorits/IMG_5611.JPG
2021-02-18 09:19:46,332 : views.py : get : 1582 : ERROR : An Error occured
Traceback (most recent call last):
  File "/code/api/views.py", line 1579, in get
    scan_photos(request.user, job_id)
  File "/code/api/directory_watcher.py", line 234, in scan_photos
    walk_directory(user.scan_directory, ps)  # now walk with photo-scannning
  File "/code/api/directory_watcher.py", line 175, in walk_directory
    walk_directory(fpath, callback)
  File "/code/api/directory_watcher.py", line 177, in walk_directory
    callback.read_file(fpath)
  File "/code/api/directory_watcher.py", line 206, in read_file
    handle_new_image(self.user, path, self.job_id)
  File "/code/api/directory_watcher.py", line 109, in handle_new_image
    photo._generate_captions()
  File "/code/api/models/photo.py", line 113, in _generate_captions
    res_places365 = inference_places365(image_path, confidence)
  File "/code/api/places365/places365.py", line 96, in inference_places365
    model = load_model()
  File "/code/api/places365/places365.py", line 83, in load_model
    model = wideresnet.resnet18(num_classes=365)
  File "/code/wideresnet.py", line 161, in resnet18
    model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs)
  File "/code/wideresnet.py", line 108, in __init__
    self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
  File "/code/wideresnet.py", line 133, in _make_layer
    layers.append(block(self.inplanes, planes))
  File "/code/wideresnet.py", line 29, in __init__
    self.conv1 = conv3x3(inplanes, planes, stride)
  File "/code/wideresnet.py", line 20, in conv3x3
    return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
  File "/miniconda/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 410, in __init__
    super(Conv2d, self).__init__(
  File "/miniconda/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 83, in __init__
    self.reset_parameters()
  File "/miniconda/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 86, in reset_parameters
    init.kaiming_uniform_(self.weight, a=math.sqrt(5))
  File "/miniconda/lib/python3.8/site-packages/torch/nn/init.py", line 384, in kaiming_uniform_
    return tensor.uniform_(-bound, bound)
  File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)
SystemExit: 1
2021-02-18 09:20:27,124 : views.py : list : 841 : INFO : querying & serializing took 0.01 seconds

@kaushikray1 This error is just the search trying to find examples, but getting something that is empty instead. This should not be the reason why the scanning stops. I will look into it anyway.
Your error is probably a different error to what has been reported a couple posts above. Could you look if a gunicorn worker is timing out? Should look something like this: [CRITICAL] WORKER TIMEOUT (pid:150),

Thank you for getting back, i looked through the log but i didn't see any item with the above mentioned error. I will also try to mount the drive with docker like you suggested and see if i have a better luck. Thank you again.

no luck here, i tried a debian machine, with docker build 20.10.2 and i get the same issue.

anything else i could test ?

Similar case. Set up containers, and the scanner consistently dies after short mount of time, with a large library on CIFS, or NFS (tried both). No apparent crash logging. Doesn't have the issue if the library is local.
Other containers not having issues with content on the same shares.

I see a similar issue with CIFS/SMB mount. Scanner dies after scanning a few pictures at a time. My setup is in a Ubuntu 18.04 server VM with, all data stored in a nas with a CIFS mount.
I haven't really seen the code but could it be due to some sort of time out? As the access time/latency to the network files are longer than the local files.
Get Outlook for Androidhttps://aka.ms/ghei36

From: Jason Hall notifications@github.com
Sent: Monday, February 22, 2021 1:42:25 PM
To: LibrePhotos/librephotos librephotos@noreply.github.com
Cc: Kaushik Ray ray.1_1@live.com; Mention mention@noreply.github.com
Subject: Re: [LibrePhotos/librephotos] NFS Issues / worker stops after one file, and does not restart (#167)
Similar case. Set up containers, and the scanner consistently dies after short mount of time, with a large library on CIFS, or NFS (tried both). No apparent crash logging. Doesn't have the issue if the library is local.
Other containers not having issues with content on the same shares.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/LibrePhotos/librephotos/issues/167#issuecomment-783587295, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABGA5GQSHPVUZNEQOZDBXBDTAKQRDANCNFSM4XO34PQA.

Just tested and was able to mount a SMB (Centos 8) share with CIFS on Ubuntu 20.04 though fstab and preform a full scan with out issue. I am going to spin up a VM (Virtual Box) next and test there. @kaushikray1 @ma-karai in your VMs how do you have your network set up NAT, Bridged ect?

I use proxmox to setup this vm: here are the settings

relevant interface configs:
docker0   Link encap:Ethernet  HWaddr 02:42:6C:97:5E:40
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr DA:77:FE:7A:67:8B
          inet addr:192.168.0.23  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::d877:feff:fe7a:678b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10338051 errors:0 dropped:0 overruns:0 frame:0
          TX packets:875584 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10451367291 (9.7 GiB)  TX bytes:572907783 (546.3 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:140 (140.0 B)  TX bytes:140 (140.0 B)

and my fstab
nas.internal.mydomain.com:/mnt/private/data/sorted-data/photos /mnt/photos nfs4 _netdev,vers=4.1 0 0

@DigitalWarrior what is your fstab configuration? I've had the same problem and can't seem to get it to scan very far without having to restart it constantly.

Seeing the exact same problem scan stops after addi g 25 to 30 pics.
Fstab setting for me is
//CIFS network path /home/user/pictures cifs username=samba_user,password=samba_password 0 0
That's pretty much it.
Get Outlook for Androidhttps://aka.ms/ghei36

From: Justin notifications@github.com
Sent: Thursday, February 25, 2021 10:34:42 PM
To: LibrePhotos/librephotos librephotos@noreply.github.com
Cc: Kaushik Ray ray.1_1@live.com; Mention mention@noreply.github.com
Subject: Re: [LibrePhotos/librephotos] NFS Issues / worker stops after one file, and does not restart (#167)
@DigitalWarriorhttps://github.com/DigitalWarrior what is your fstab configuration? I've had the same problem and can't seem to get it to scan very far without having to restart it constantly.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/LibrePhotos/librephotos/issues/167#issuecomment-786387983, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABGA5GXNDFCJIE2JHISSXOLTA4JFFANCNFSM4XO34PQA.

Server is Cent os 8 running Samba version 4.12.3
On machine machine running LibrePhotos the fstab mount is //192.168.1.9/NAS/testpics /media/testpics cifs user=USERNAME,password=PASSWORD,iocharset=utf8,vers=3.0,noperm 0 0

scanning got stuck
I use a NAS for hosting my pictures (and thumbnails):
HEre is my fstab:
192.168.0.14:/nfs/Public/DATA/DomoBase/Librephotos /home/pascal/Librephotos/media/ nfs defaults,_netdev 0 0
192.168.0.14:/nfs/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures nfs defaults,_netdev 0 0

Is there any workaround or coming release solving such problem. Sounds basic to me (but I may-be wrong) to use a mount point for photos...

Hey I installed a nfs client and a cifs client within the docker image. Also the dependency for generating images is now switched. Could somebody try it again?

Will give a try... just to be sure: a stop, a pull and a up -d is enough?

Yes that should be enough

I did it.
It is slow so I cannot be 100% sure but it seems stuck at 269/34778.

I reload a worker to be sure and the funny thing is that the total count of pictures is not the same between the two:

the 2nd worker is now stuck at 312... so @derneuere it seems the bug is not solved

Can you give last log line?

To be honest, I never found useful log with Librephotos (by useful, I mean one that make you have an idea where the problem is).
Here is the docker-compose logs last lines (but if you want something else, tell me):
proxy_1     | 192.168.0.90 - - [22/Mar/2021:21:06:19 +0000] "GET /api/rqavailable/ HTTP/1.1" 200 428 "http://192.168.0.25:3000/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
proxy_1     | 192.168.0.90 - - [22/Mar/2021:21:06:22 +0000] "GET /api/rqavailable/ HTTP/1.1" 200 428 "http://192.168.0.25:3000/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
proxy_1     | 2021/03/22 21:06:22 [info] 28#28: *4684 client 192.168.0.90 closed keepalive connection
proxy_1     | 192.168.0.90 - - [22/Mar/2021:21:06:24 +0000] "GET /api/rqavailable/ HTTP/1.1" 200 428 "http://192.168.0.25:3000/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
proxy_1     | 192.168.0.90 - - [22/Mar/2021:21:06:25 +0000] "GET /api/rqavailable/ HTTP/1.1" 401 185 "http://192.168.0.25:3000/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
proxy_1     | 192.168.0.90 - - [22/Mar/2021:21:06:25 +0000] "POST /api/auth/token/refresh/ HTTP/1.1" 200 434 "http://192.168.0.25:3000/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
proxy_1     | 192.168.0.90 - - [22/Mar/2021:21:06:26 +0000] "GET /api/rqavailable/ HTTP/1.1" 200 428 "http://192.168.0.25:3000/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
proxy_1     | 2021/03/22 21:06:32 [info] 28#28: *4690 client 192.168.0.90 closed keepalive connection
proxy_1     | 192.168.0.90 - - [22/Mar/2021:21:07:26 +0000] "GET /api/rqavailable/ HTTP/1.1" 200 428 "http://192.168.0.25:3000/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
proxy_1     | 2021/03/22 21:07:32 [info] 28#28: *4695 client 192.168.0.90 closed keepalive connection

/logs/gunicorn_django.log

Unauthorized: /api/rqavailable/
Unauthorized: /api/rqavailable/
Unauthorized: /api/rqavailable/
Unauthorized: /api/rqavailable/
Unauthorized: /api/rqavailable/
Unauthorized: /api/rqavailable/
Not Found: /api/jobs/
Internal Server Error: /api/searchtermexamples/
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/code/api/views.py", line 1360, in get
    search_term_examples = get_search_term_examples(request.user)
  File "/code/api/api_util.py", line 163, in get_search_term_examples
    term_thing = random.choice(datum['things'])
  File "/usr/local/lib/python3.8/random.py", line 290, in choice
    raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence

This error is not on the scanning process

I have some log like that
2021-03-19 02:22:25,611 : directory_watcher.py : handle_new_image : 87 : INFO : job f4929428-a1b4-4e79-9755-e21d5dfbf993: handling image /var/lib/librephotos/data/photos/2017/07-MariageAngouleme/PhotoPhographe/publication/claire_thomas0009.png
2021-03-19 02:22:27,555 : photo.py : _generate_captions : 124 : INFO : generated places365 captions for image /var/lib/librephotos/protected_media/thumbnails_big/4a854b6c20e9ec16799820578845fa071.jpg.
2021-03-19 02:22:31,228 : photo.py : _extract_faces : 346 : INFO : image 1e788dae3cf0009599ba27baf92a20821: 1 face(s) saved

Ownphotos.log may-be?

2021-03-22 18:23:38,895 : directory_watcher.py : handle_new_image : 129 : ERROR : job ba106f76-673a-47f1-8d7e-4ea0468c7c0c: could not load image /data/2011/2011-03-05 Queyras/P1050256.JPG. reason: [Errno 37] No locks available
Traceback (most recent call last):
  File "/code/api/directory_watcher.py", line 100, in handle_new_image
    photo._generate_thumbnail()
  File "/code/api/models/photo.py", line 138, in _generate_thumbnail
    self.thumbnail_big.save(self.image_hash + '.jpg', ContentFile(transfer.getvalue()))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 270, in _save
    locks.lock(fd, locks.LOCK_EX)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/locks.py", line 110, in lock
    ret = fcntl.flock(_fd(f), flags)
OSError: [Errno 37] No locks available
2021-03-22 18:23:41,876 : directory_watcher.py : handle_new_image : 78 : INFO : job ba106f76-673a-47f1-8d7e-4ea0468c7c0c: handling image /data/2011/2011-03-05 Queyras/P1050260.JPG
2021-03-22 18:23:59,933 : directory_watcher.py : handle_new_image : 129 : ERROR : job ba106f76-673a-47f1-8d7e-4ea0468c7c0c: could not load image /data/2011/2011-03-05 Queyras/P1050260.JPG. reason: [Errno 37] No locks available
Traceback (most recent call last):
  File "/code/api/directory_watcher.py", line 100, in handle_new_image
    photo._generate_thumbnail()
  File "/code/api/models/photo.py", line 138, in _generate_thumbnail
    self.thumbnail_big.save(self.image_hash + '.jpg', ContentFile(transfer.getvalue()))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 270, in _save
    locks.lock(fd, locks.LOCK_EX)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/locks.py", line 110, in lock
    ret = fcntl.flock(_fd(f), flags)
OSError: [Errno 37] No locks available
2021-03-22 18:24:03,380 : directory_watcher.py : handle_new_image : 78 : INFO : job ba106f76-673a-47f1-8d7e-4ea0468c7c0c: handling image /data/2011/2011-03-05 Queyras/P1050280.JPG
2021-03-22 21:47:07,013 : api_util.py : get_location_clusters : 257 : INFO : location clustering took 2.51 seconds

Ok this error is issue with lock NFS deamon.
What is your NFS server? how you mount nfs on docker?

Here is the fstab of the server running the docker (Ubuntu 29.04):
192.168.0.14:/nfs/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures nfs defaults,_netdev 0 0

ubuntu 29.04 you are in the future? lol
have you some log like commande "dmesg" kernel log or syslog

ubuntu 29.04 you are in the future? lol

yeah :) I edited when I saw it :)

have you some log like commande "dmesg" kernel log or syslog

doesn't seem to work in the docker container

Few comments:

I started again the worker and the "no lock" message appears since the beginning so why it is getting stuck after tens of files? but in any case, it doesn't sound good to have such errors
why do we need to lock the file if we just read it??

Libre photos project just call save image via a models field
After it dango who save file with lock.
It's not a librephotos issue
Can you test (maybe your server do not support lock)
192.168.0.14:/nfs/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures nfs vers=3,nolock,_netdev 0 0

Restart and give process log

Sorry for the delay but it takes a lot of time between the time the webpage is accessible and the time we can effectively login (before there is cannot connect with backend message)
Anyway, the error is the same:
OSError: [Errno 37] No locks available
2021-03-23 09:53:12,549 : directory_watcher.py : rescan_image : 149 : ERROR : job 56842e8a-4ab5-41db-8e38-9275c8dbe3ef: could not load image /data/2011/IMG_0850.PNG.$.jpg. reason: [Errno 37] No locks available
Traceback (most recent call last):
  File "/code/api/directory_watcher.py", line 144, in rescan_image
    photo._generate_thumbnail()
  File "/code/api/models/photo.py", line 182, in _generate_thumbnail
    self.square_thumbnail_small.save(self.image_hash + '.jpg', ContentFile(transfer.getvalue()))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 270, in _save
    locks.lock(fd, locks.LOCK_EX)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/locks.py", line 110, in lock
    ret = fcntl.flock(_fd(f), flags)
OSError: [Errno 37] No locks available

nfs server is? a nas?

Yes, WD my cloud to be precise... never got any problem with it and many mount I am doing

Le 23 mars 2021 à 22:44, tomamplius @.***> a écrit :

nfs server is? a nas?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

the issue is https://code.djangoproject.com/ticket/9400.
But normaly WD my cloud use nfs v4

It is an old one so may-be not... I will check later on
If it is linked to nfs, mounting with cifs should work? If yes, I can give a try

Le 23 mars 2021 à 23:14, tomamplius @.***> a écrit :

the issue is https://code.djangoproject.com/ticket/9400.
But normaly WD my cloud use nfs v4
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

I work with cifs without issue

Good to know
Will test it tomorrow and report

Le 23 mars 2021 à 23:18, tomamplius @.***> a écrit :

I work with cifs without issue
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

but  error "[Errno 37] No locks available" is a no-blocking issue. Exception is catch and image is not processed.
If the process is lock then it's an other issue

By the way I checked and nfs used is version 3

Le 23 mars 2021 à 23:18, tomamplius @.***> a écrit :

I work with cifs without issue
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

test before with lock option
192.168.0.14:/nfs/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures nfs vers=3,lock,_netdev 0 0

Well there is not a max of exception? Cause it makes error with the first
files but is stuck after tens of it without other messages
Will give a try tomorrow with cifs (I just chose nfs thinking it was faster
but I use cifs on other mounts/machine)
Le mar. 23 mars 2021 à 23:20, tomamplius @.***> a
écrit :

but error "[Errno 37] No locks available" is a no-blocking issue.
Exception is catch and image is not processed.
If the process is lock then it's an other issue
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/LibrePhotos/librephotos/issues/167#issuecomment-805304671,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABWMPITIETFP5RAQFN2AGB3TFEH3RANCNFSM4XO34PQA
.

Ok, if it helps I will give a try with lock option
Le mar. 23 mars 2021 à 23:21, tomamplius @.***> a
écrit :

test before with lock option
192.168.0.14:/nfs/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures nfs vers=3,lock,_netdev 0 0
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/LibrePhotos/librephotos/issues/167#issuecomment-805305704,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABWMPITPW5HLOXRD7R23KCLTFEH7PANCNFSM4XO34PQA
.

The fact that statd is missing as described in the django bug report is
something that I have seen in many places while looking for this locking
problem... may-be statd is missing in the container??
Le mar. 23 mars 2021 à 23:14, tomamplius @.***> a
écrit :

the issue is https://code.djangoproject.com/ticket/9400.
But normaly WD my cloud use nfs v4
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/LibrePhotos/librephotos/issues/167#issuecomment-805301738,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABWMPIREBQIVBLOKKZPRXL3TFEHEZANCNFSM4XO34PQA
.

test before with lock option and you need start statd daemon
192.168.0.14:/nfs/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures nfs vers=3,lock,_netdev 0 0

Same errors happen.
Then I tested by mounting with CIFS
//192.168.0.14/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures cifs username=guest,password=guest,_netdev 0 0
I got
2021-03-24 09:34:50,522 : directory_watcher.py : is_valid_media : 22 : INFO : Could not handle /data/2011/IMG_0742.MOV, because delegate failed 'ffmpeg' -nostdin -v -1 -i '%i' -vframes %S -vcodec pam -an -f rawvideo -y '%u.pam' 2> '%u'' @ error/delegate.c/InvokeDelegate/1928`
and then the same error than before appeared:

2021-03-24 09:38:37,304 : directory_watcher.py : rescan_image : 149 : ERROR : job 218c61a7-1aac-4f35-8829-3e98f3fb8485: could not load image /data/2011/2011-03-05 Queyras/P1050253.JPG. reason: [Errno 37] No locks available
Traceback (most recent call last):
  File "/code/api/directory_watcher.py", line 144, in rescan_image
    photo._generate_thumbnail()
  File "/code/api/models/photo.py", line 182, in _generate_thumbnail
    self.square_thumbnail_small.save(self.image_hash + '.jpg', ContentFile(transfer.getvalue()))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 270, in _save
    locks.lock(fd, locks.LOCK_EX)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/locks.py", line 110, in lock
    ret = fcntl.flock(_fd(f), flags)
OSError: [Errno 37] No locks available

any other thing to try?

The fact that statd is missing as described in the django bug report is something that I have seen in many places while looking for this locking problem... may-be statd is missing in the container?? Le mar. 23 mars 2021 à 23:14, tomamplius @.***> a écrit :
…
the issue is https://code.djangoproject.com/ticket/9400. But normaly WD my cloud use nfs v4 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#167 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABWMPIREBQIVBLOKKZPRXL3TFEHEZANCNFSM4XO34PQA .

You could look into the backend container and look if statd is running.

You could look into the backend container and look if statd is running

ps -aef | grep statd show nothing...

Then start it in the backend container and look if scanning then works.

Then start it in the backend container and look if scanning then works.

root@c86f197efcfa:/code# service statd start
statd: unrecognized service

Can you check and give result?
apt-cache policy nfs-common

I am going to pull latest version and test

Outside docker:
nfs-common:
  Installé : 1:1.3.4-2.5ubuntu3.3
  Candidat : 1:1.3.4-2.5ubuntu3.3
 Table de version :
 *** 1:1.3.4-2.5ubuntu3.3 500
        500 http://fr.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.3.4-2.5ubuntu3 500
        500 http://fr.archive.ubuntu.com/ubuntu focal/main amd64 Packages

Inside docker backend:
nfs-common:
  Installed: 1:1.3.4-2.5+deb10u1
  Candidate: 1:1.3.4-2.5+deb10u1
  Version table:
 *** 1:1.3.4-2.5+deb10u1 500
        500 http://deb.debian.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status

@darrepac tkx
I have create minimal script python to reproduce and help us to understant the issue
I have create ubuntu 20.04 env to test with nfs export and mount point.
The script work. I want to understant if the issue is on export (server) side or if is on docker side?
Test script can be call with create file in parameter
sample call
python3 test.py /tmp/testfile

Can you test on other docker to confirm that not work only on librephotos docker in first
#!/usr/bin/python3
import fcntl

def _fd(f):
    """Get a filedescriptor from something which could be a file or an fd."""
    return f.fileno() if hasattr(f, 'fileno') else f

def lock(f, flags):
    ret = fcntl.flock(_fd(f), flags)
    return ret == 0

def unlock(f):
    ret = fcntl.flock(_fd(f), fcntl.LOCK_UN)

with open('./file', 'w') as f:
    lock(f, fcntl.LOCK_EX)
    f.write('test')
    unlock(f)

inside backend docker
python3 test.py /tmp/testfile => no error
python3 test.py /data/testfile => no error

outside backend (=server)
python3 test.py /tmp/testfile => no error
python3 test.py pictures/testfile => error!
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    with open(sys.argv[1], 'w') as f:
IOError: [Errno 13] Permission denied: 'pictures/testfile'

nfs is mount on /data?

In the server, nfs is mount on « pictures
Le dim. 28 mars 2021 à 19:05, tomamplius @.***> a
écrit :

nfs is mount on /data?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LibrePhotos/librephotos/issues/167#issuecomment-808925431,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABWMPIWZOLTVPV6WPNK4LOLTF5VVVANCNFSM4XO34PQA
.

sorry i don't understand.
You need to execute python3 test.py /path/into/nfs/share inside backend docker?
outside backend = your computer/ docker server?
Permission denied: 'pictures/testfile' means you have not right to write

Sorry
To make it clear, my pictures are on a NFS mount (or CIFS depending the test...previous test above was with CIFS)
(NAS: 192.168.0.14):
192.168.0.14:/nfs/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures nfs defaults,_netdev 0 0
outside docker:inside docker
/home/pascal/Librephotos/pictures:/data

Then on the server side (=outside backend docker):
python3 test.py /home/pascal/Librephotos/pictures/testfile
With NFS mount:
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    lock(f, fcntl.LOCK_EX)
  File "test.py", line 10, in lock
    ret = fcntl.flock(_fd(f), flags)
OSError: [Errno 37] No locks available

On the docker side
python3 test.py /data/testfile
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    lock(f, fcntl.LOCK_EX)
  File "test.py", line 10, in lock
    ret = fcntl.flock(_fd(f), flags)
OSError: [Errno 37] No locks available

So it seems that the error can be reproduced with NFS mount (for CIFS, I need to check again)

Ok, it's means that your NFS server (Nas) doesn't allow lock

Nas cannot be updated and is in nfs version 3...
Let's look back on CIFS mount.
I solved the permission issue, contrary to NFS mount I have no error with:
python3 test.py /home/pascal/Librephotos/pictures/testfile
But on the librephoto log side, I end up with:
2021-03-28 21:45:26,042 : directory_watcher.py : rescan_image : 141 : ERROR : job 871824ac-4767-45c2-ba63-42544caa5777: could not load image /data/2011/2011-03-05 Queyras/P1050263.JPG. reason: [Errno 37] No locks available
Traceback (most recent call last):
  File "/code/api/directory_watcher.py", line 136, in rescan_image
    photo._generate_thumbnail(False)
  File "/code/api/models/photo.py", line 183, in _generate_thumbnail
    self.square_thumbnail_small.save(self.image_hash + '.jpg', ContentFile(transfer.getvalue()))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 270, in _save
    locks.lock(fd, locks.LOCK_EX)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/locks.py", line 110, in lock
    ret = fcntl.flock(_fd(f), flags)
OSError: [Errno 37] No locks available

May-be a stupid question: why is there a locking mechanism? can't we have the source pictures read-only??

It would be great if you could follow the steps we suggest instead of trying something different.
To explain it simply: We don't only use python, we also use django. Django handles file locking automatically when you use it. Django needs statd to remember the locks I think. We create thumbnails and save them to a directory. You save the thumbnails to your NFS share and not locally. When you create a file on a NFS share you have to handle the file locking.
I added statd service to the entrypoint.sh file. Will soon create a new docker image.

It would be great if you could follow the steps we suggest instead of trying something different.

??
I found this remark a bit harsh with my time spent on trying to help you to find where the problem is (a problem that I never faced with any other projects).
I think I have done all tests requested and even more, and spent quite some time on it. I don't have problem about it as it is my contribution to help this nice open source project, but if it is to receive such remark better to be a consumer...

I didn't mean to be harsh 😅
We currently have as open options to activate statd within the docker which should be included with the new docker image. You could also try to save the thumbnails locally.

I will give a try to the new docker image once you publish it
Having the thumbnails stored locally is also an option (but it would depend on the disk size it takes, I would check)

Locking mechanism is here because media file is link with a database fields and it's a way to prevent concurrent access.
Can you re-explain why you don't use CIFS? Right access? "IOError: [Errno 13] Permission denied: 'pictures/testfile'"
how you mount the FS?

Locking mechanism is here because media file is link with a database fields and it's a way to prevent concurrent access.
Can you re-explain why you don't use CIFS? Right access? "IOError: [Errno 13] Permission denied: 'pictures/testfile'"
how you mount the FS?

As stated in few comments above, I solved my right access error (was missing uid/gid in mounting) and tried with CIFS and end up with the same locking issue (even if the python test program didn't make any error this time).
Cifs vs nfs? I don't care to use one or another (I thought NFS was more efficient, but no problem to use CIFS... I already tried both)
How I mount CIFS? Here is an extract of my fstab:
//192.168.0.14/Public/DATA/C/DATA/Photos/Photos_canon /home/pascal/Librephotos/pictures cifs username=guest,password=guest,uid=pascal,gid=pascal,_netdev 0 0

Ok sorry for my misunderstand (i not good in english)

Ok sorry for my misunderstand (i not good in english)

On peut le faire en français si tu veux ;)

I didn't mean to be harsh 😅
We currently have as open options to activate statd within the docker which should be included with the new docker image. You could also try to save the thumbnails locally.

I am back after all those pytorch  / avx issue.
Statd: here is what I have in the logs:
backend_1   | statd: unrecognized service
Thumbnail in local: already the case, doesn't change anything.

Hey People who follow this issue. I tried https://github.com/photoview/photoview which seems to be also based on the same source. I dont have these issues there, maybe someone can check what the difference is.

@ma-karai Hey, quick questions, on which host systems are you running your VMs? Does it use a linux system or do you use macOS/Windows?

I am using ubuntu server 20.04. Thank you.
Get Outlook for Androidhttps://aka.ms/ghei36

From: Niaz Faridani-Rad @.>
Sent: Monday, November 22, 2021 1:26:53 PM
To: LibrePhotos/librephotos @.>
Cc: Kaushik Ray @.>; Mention @.>
Subject: Re: [LibrePhotos/librephotos] NFS Issues / worker stops after one file, and does not restart (#167)
@ma-karaihttps://github.com/ma-karai Hey, quick questions, on which host systems are you running your VMs? Does it use a linux system or do you use macOS/Windows?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/LibrePhotos/librephotos/issues/167#issuecomment-975802904, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABGA5GVA4B37ZC4AQEPCEKTUNKDO3ANCNFSM4XO34PQA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@ma-karai Hey, quick questions, on which host systems are you running your VMs? Does it use a linux system or do you use macOS/Windows?

My hypervisor is debian (latest proxmox) based and i tried different VM OSs, currently I am running on Alpine (latest)

It's been a while since anyone reported on this, but I have the same problem. Host system is a Synology DS1815+ and the photos are on a NAS, mounted using CIFS or NFS (both don't work in the same manner). It will scan 1 image and stale. If we close the worker and restart it, it will crash. It means that Librephotos is unusuable for me, which is a shame, because it looks very promising.
I've catched the logs during the stalling and it looks like this (filenames are changed because of personal data):
development backend starting
[2022-03-31 11:27:49 +0000] [153] [INFO] Starting gunicorn 20.1.0
[2022-03-31 11:27:49 +0000] [153] [INFO] Listening at: http://0.0.0.0:8001 (153)
[2022-03-31 11:27:49 +0000] [153] [INFO] Using worker: gevent
[2022-03-31 11:27:49 +0000] [155] [INFO] Booting worker with pid: 155
[2022-03-31 11:27:49 +0000] [156] [INFO] Booting worker with pid: 156
/usr/local/lib/python3.9/dist-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/usr/local/lib/python3.9/dist-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/usr/local/lib/python3.9/dist-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/usr/local/lib/python3.9/dist-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/usr/local/lib/python3.9/dist-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/usr/local/lib/python3.9/dist-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/usr/local/lib/python3.9/dist-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
11:28:02 Worker rq:worker:337076be197e49578b4f0aa9ae2f2e9a: started, version 1.6.1
11:28:02 Subscribing to channel rq:pubsub:337076be197e49578b4f0aa9ae2f2e9a
11:28:02 *** Listening on default...
11:28:02 Cleaning registries for queue: default
11:28:02 default: api.apps.build_index() (be85746f-df2e-4d0e-bb8a-faf0f910d8a1)
127.0.0.1 - - [2022-03-31 11:28:03] "POST /build/ HTTP/1.1" 200 139 0.112014
11:28:03 default: Job OK (be85746f-df2e-4d0e-bb8a-faf0f910d8a1)
11:28:03 Result is kept for 60 seconds
11:28:03 default: api.apps.build_index() (fc3cd9df-12a1-4912-8f44-abd8e01f2655)
127.0.0.1 - - [2022-03-31 11:28:03] "POST /build/ HTTP/1.1" 200 139 0.002356
11:28:03 default: Job OK (fc3cd9df-12a1-4912-8f44-abd8e01f2655)
11:28:03 Result is kept for 60 seconds
11:28:03 default: api.apps.build_index() (a16660b9-8dab-4bd4-86f2-0d621ad33293)
127.0.0.1 - - [2022-03-31 11:28:03] "POST /build/ HTTP/1.1" 200 139 0.001903
11:28:03 default: Job OK (a16660b9-8dab-4bd4-86f2-0d621ad33293)
11:28:03 Result is kept for 60 seconds
11:28:03 default: api.apps.build_index() (03490e84-651e-4f76-96ff-d6e1e8b5bf63)
127.0.0.1 - - [2022-03-31 11:28:03] "POST /build/ HTTP/1.1" 200 139 0.002409
11:28:03 default: Job OK (03490e84-651e-4f76-96ff-d6e1e8b5bf63)
11:28:03 Result is kept for 60 seconds
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/jobs/
Unauthorized: /api/dirtree/
Unauthorized: /api/jobs/
Unauthorized: /api/albums/date/list/
Unauthorized: /api/dirtree/
Not Found: /api/jobs/
Not Found: /api/jobs/
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/
11:28:34 default: api.directory_watcher.scan_photos(<User: admin>, False, UUID('b61d265a-4f0c-4e4e-bf29-0bf4a674ff93')) (b1112090-d91a-4590-807f-2e823c973599)
Not Found: /api/jobs/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/jobs/
Unauthorized: /api/dirtree/
Unauthorized: /api/jobs/
Not Found: /api/jobs/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/jobs/
Unauthorized: /api/dirtree/
Unauthorized: /api/jobs/
Not Found: /api/jobs/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/dirtree/
Unauthorized: /api/jobs/
Unauthorized: /api/jobs/
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/
Not Found: /api/jobs/
Not Found: /api/jobs/
INFO:ownphotos:job b61d265a-4f0c-4e4e-bf29-0bf4a674ff93: handling image /data/Photos/Augustus 2017/natuur-exploitatie.jpg
ERROR:ownphotos:job b61d265a-4f0c-4e4e-bf29-0bf4a674ff93: could not load image /data/Photos/Augustus 2017/IMG-20170827-WA0003.jpg. reason: unsupported operand type(s) for /: 'NoneType' and 'NoneType'
Traceback (most recent call last):
INFO:ownphotos:Old Album Date: 2017-05-08
INFO:ownphotos:New Album Date: 2017-05-08
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/jobs/
Unauthorized: /api/dirtree/
Unauthorized: /api/jobs/
Not Found: /api/jobs/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/jobs/
Unauthorized: /api/dirtree/
Unauthorized: /api/jobs/
Not Found: /api/jobs/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/dirtree/
Unauthorized: /api/jobs/
Unauthorized: /api/jobs/
Not Found: /api/jobs/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/jobs/
Unauthorized: /api/jobs/
Unauthorized: /api/dirtree/
Not Found: /api/jobs/
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/
Not Found: /api/jobs/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/searchtermexamples/
Unauthorized: /api/jobs/
Unauthorized: /api/dirtree/
Unauthorized: /api/jobs/
Not Found: /api/jobs/
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/

Pretty sure it's not network share related.
I'm using Synology too and got the same issue.
But i simply made a new folder and placed a few pictures inside.

i actually do have write access as i've uploaded a test image and that worked!
But still, the scan is stuck.
Here is a log from the backend.
backend.csv
No matter what i try i only see "No images found" and whenever i remove the scan and start a new one in user settings it gets stuck.

Just chiming in to say that the issue with NFS mounts continues to persists. I just recently discovered librephotos and I really love the app but that fact that scans seem to stall after a couple of pictures makes it unusable to me.
Tried all the different ways of mounting the NFS share suggested in this thread but it didn't help.

Since the beginning of LP, I have always used an NFS Share and it has always worked perfectly for me!

The culprit are probably non-responsive NFS shares. It may hang or experience network issues, and we do not handle potential timeouts and failures gracefully. We should probably write functions for the os.path libraries to add timeouts and retries.

Use timeouts: Wrap your file operations in timeouts to prevent indefinite hanging. The os module doesn't provide built-in timeout functionality, but you can use the signal module to achieve this. Here's an example:
import signal

def handle_timeout(signum, frame):
    raise TimeoutError("File operation timed out")

# Set a timeout of 5 seconds
signal.signal(signal.SIGALRM, handle_timeout)
signal.alarm(5)

try:
    # Perform your file operations here
    exists = os.path.exists(file_path)
    # Continue with your code if the operation succeeds
except TimeoutError:
    # Handle the timeout error
    # Log the error, retry, or perform any necessary cleanup

finally:
    # Disable the alarm
    signal.alarm(0)

This way, if the file operation takes longer than the specified timeout, a TimeoutError will be raised, and you can handle it accordingly.

Implement retries: In addition to timeouts, you can implement retry logic to make multiple attempts to perform the file operation before giving up. Here's an example using the tenacity library:
from tenacity import retry

# Decorate the function with retrying options
@retry(stop_max_attempt_number=3, wait_fixed=2000)
def perform_file_operation():
    exists = os.path.exists(file_path)
    # Continue with your code if the operation succeeds

try:
    perform_file_operation()
except retrying.RetryError:
    # Handle the retry error
    # Log the error, perform any necessary cleanup, or raise a custom exception

In this example, the perform_file_operation function will be retried up to three times with a fixed delay of 2 seconds between retries. Adjust the retry options according to your specific needs.

Implement error handling and logging: Wrap your file operations in appropriate try-except blocks to catch any potential exceptions and handle them gracefully. You can log the errors and take appropriate action based on the specific exception raised. For example:
try:
    exists = os.path.exists(file_path)
    # Continue with your code if the operation succeeds
except FileNotFoundError:
    # Handle the file not found error
    # Log the error, retry, or perform any necessary cleanup
except OSError as e:
    # Handle other OS-level errors
    # Log the error, retry, or perform any necessary cleanup
except Exception as e:
    # Handle any other unexpected exceptions
    # Log the error, retry, or perform any necessary cleanup

By catching and handling specific exceptions, you can provide better feedback and error handling for different scenarios.

authors: DigitalWarrior Jayceh Nerodacles darrepac derneuere haldi4803 ituri kaushikray1 kicker334 ma-karai riki137 selfhosted-heaven tomamplius

repo: LibrePhotos/librephotos


text:
instarepl broken

I was going through the tutorial for instarepl and I found that it is broken
Environment - Ubuntu 14.04 64-bit
Light Table version - 0.7.2
Binary version - 0.8.6

In the image if I change the (+ x y) line then the numbers instead of changing increase. I mean it is like this. See the below image? There are a lot more 3's present then there should be.

Also it seems the colouring scheme that was mentioned in the video tutorial is not working either.

I thought that this was a really cool thing that I can present to some people but seeing the instarepl broken is a complete bummer.
I am learning clojure so probably going to use this till I can find time for Emacs. I wish this was not broken  as I seriously think that this is really nice.

This is fixed in the latest clojure plugin (0.1.3 currently).

ctrl + space  (open command bar)
"Plugins: Show plugin manager"
Find the Clojure plugin and click upgrade

After it's installed, unfortunately you current need to reload it's behaviors or restart LT
Reloading:

ctrl + space
"App - Reload behaviors"

Once that is done you can open an instarepl and results shouldn't be stacking any more

Thanks for the details. Different colors not working. But maybe that is not important.

The video's are getting pretty dated,
Bug reports really need to state what is wrong rather than referring to tutorials for us to be able to consider them. (on a side note, the tutorials page probably needs a disclaimer that things might have changed in later version, alternatively we remove them altogether and refer to them as external non supported learning resources).
Closing this issue. Let me know if 0.1.3 doesn't solve the issue with results-stacking.

"Find the Clojure plugin and click upgrade"
This would be really cool, if there were an "upgrade" to click on.
When I open the plugin manager and I hover over Clojure 0.1.0, I get a little popup menu on the right, with the choices "uninstall" and "source".  I wonder where I might find an "upgrade" to click on.

@kleinfelter Please create a new issue for your problem.

source: gharchive/issue

authors: anshbansal kenny-evitt kleinfelter rundis

repo: LightTable/LightTable

license: MIT

license_type: permissive

license_source: github-api


text:
onUpdate

How do I specify an onUpdate handler ? It did not help adding onUpdate as a function (taking item, callback as arguments) in options passed to Timeline.

@prabuvenkat Is this like what you are doing already?  If so, what is the issue you are facing?  In order for onUpdate to fire, you must make sure that editable is set to true per the referenced documentation.
import React, {Component} from 'react';
import Timeline from 'react-visjs-timeline';

export default class App extends Component {
  constructor(props) {
    super(props);

    this.state = {
      options: {
        editable: true,
        onUpdate: (item, callback) => {
          // implement your onUpdate actions here, or you could reference another function
          // see <http://visjs.org/docs/timeline/#Editing_Items>
        }
      },
      items: [
        { start: '2017-01-01', end: '2017-01-31', content: 'January' },
        { start: '2017-02-01', end: '2017-02-28', content: 'February' }
      ]
    };
  }

  render() {
    return (
      <Timeline {...this.state}/>
    );
  }
}

Hei!
Yes, this is exactly what I have.. but the function is never invoked. My items are of type 'range'. I am trying to get the function when I change the end/start of a block from the ui itself..
Am I wrong in assuming this is the function which will be invoked when I, say ,modify the range of one of the items ?

    selectable: true,
    editable: true,
    onUpdate: (item, callback) => {
      console.log('{update} item', item);
      console.log('{update} cb: ', callback);
      callback(item);
    },

I realize onMove is what I must be using (not onUpdate) and it works perfectly!

authors: ErinsMatthew prabuvenkat

repo: Lighthouse-io/react-visjs-timeline

license: mit

license_source: bigquery