text:
Update to compile with image v0.24

(it's basically all clippy lints all the way down)

Great job @qm3ster!

@kennytm is there any blocker to this? Thanks!

It sure would be nice to have this

@LeviSchuck you can use it as follows today. This should allegedly replace qrcode used by any dependencies as well.
[dependencies]
qrcode = "0.12"

[patch.crates-io]
qrcode = { git = "https://github.com/qm3ster/qrcode-rust", rev =  "0f75a68c7ab76c5a4ce75185a9b3367bffb7d109" }

Don't forget to use the commit hash (rev) when doing things like this, so that the untrustworthy fork owner (me, in this case) doesn't rugpull you.

( @qm3ster I am already using your fork, thank you 🙏 and good tip on the commit hash )

I'm not the maintainer, but a couple of comments on this pr.

The commit messages are severely lacking. "get it to compile" (with what?), "go hog wild" (WTH is this supposed to mean)
It would be better to keep formatting/lint changes seperate from changes related to the new version of image.

@qm3ster are you interested in maintaining the crate under a different name? seems like @kennytm has lost interest and I'd like to loose the git dependency on your PR.

Hi! In case no one's interested, I'm willing to overtake ownership of this crate. Two of my daily use tools depend on this.

I'd suggest to create a new repo with a new name, update the readme, publish a new release to crates.io and then post a link here. Thanks!

That's the final option. I am reaching out to @kennytm via mail to know if it's possible to give write access or transfer the ownership to the interested maintainer(s).

Forked and released qrencode.
GitHub: https://github.com/sayanarijit/qrencode
Crate: https://crates.io/crates/qrencode
Currently, this forked release contains dependency upgrades and minor cleanups. I'll do a proper cleanup later. Any help from the community is appreciated.

repo: kennytm/qrcode-rust


text:
It seems not working within a bootstrap site, can you confirm?

As I said, installed in a Bootstrap 3.3.5 website. I'll try to figure out something, do you have any experience about?
(!) OOOOPS, it was my bad... sorry for bad report!

Please read the contribution guidelines in the repository :)

source: gharchive/issue

authors: MigthyThor simeydotme

repo: kenwheeler/slick

license: MIT

license_type: permissive

license_source: github-api


text:
adaptiveHeight + lazyLoad does not re-expand the DIV after switching slides

Set adaptiveHeight & adaptiveHeight set true. Switch slides. The div's height is set at 1PX height.
http://jsfiddle.net/kb2nLnj1/1/

Merged PR

Nice first fix!
only thing is that now, it "jumps" instead of nicely getting bigger (or smaller) in height.

Those are all the same height

I edited my comment, wrong link. Sorry. Check again :)

@kenwheeler should I open a new ticket as this one has been closed? :)

Any news about this? :)

Temporary fix:
/**
* Adjust the height of the slider according to the active item
*/
$('.my-slider').on('init afterChange lazyLoaded', function(){
  var heightActiveSlider = $('.my-slider .item.slick-active').height();
  $('.my-slider').height(heightActiveSlider);
});

authors: Luc45 alesas kenwheeler rgbjoy


text:
Gin-Mongo & Mux-SQL

[x] Gin-mongo Updated
[x] Mux-SQL Updated

Gin-Mongo
EVERY THING IS RUNNING PROPERLY

Mux-SQL Updated
EVERY THING IS WORKING PROPERLY

@Sonichigo @re-Tick i have reviewed this pr. everything is working properly

source: gharchive/pull-request

authors: AnkitKumarvaid Sonichigo

repo: keploy/samples-go


text:
Add status from last event to artifacts list

Is your feature request related to a problem? Please describe.
All Evaluation artifacts show "Evaluation started", although all of them are done:

Describe the solution you'd like
I would be great if the last event status would be displayed in the artifact column like for example:

Colors for an evaluation with a positive outcome would be great too... otherwise only failed evaluations are marked.
-> Not management friendly 😉
I can write an additional issue if it helps to keep track.
Describe alternatives you've considered
Additional context

Thanks for proposing this improvement.
As we are going to implement the new Shipyard format (Shipard v.0.2 see here: https://github.com/keptn/enhancement-proposals/pull/6) this improvement will be taken into consideration.

Actually the bridge now shows the status of a sequence. If the sequence/task is ongoing, it shows a loading icon and the title says "Sequence started". Once the sequence is finished, it will show the icon and "Sequence succeeded" or "Sequence failed", depending on the result of the sequence.

authors: ermin-muratovic johannes-b neox5

repo: keptn/keptn


text:
keptn upgrade command: Avoid the version check via a flag

When executing keptn upgrade using a nightly version or a preview of a release branch (e.g., with 0.7.3), we always get the following message:
$ keptn upgrade
Error: No upgrade path exists from Keptn version 0.7.2 to 0.7.3

We should however provide the user the option to overwrite this, e.g., using
$ keptn upgrade --skip-upgrade-check

Definition of Done

[ ] Introduce a new parameter/flag for the upgrade command to disable the upgrade path check
[ ] Unit test added the existing and new behaviour

Is this feature still needed? If it's still required I would like to take this up @christian-kreuzberger-dtx

Is this feature still needed? If it's still required I would like to take this up @christian-kreuzberger-dtx

Yes, still needed. I'll assign it to you! Do you know where to start?

Yes, still needed. I'll assign it to you! Do you know where to start?

Thanks for assigning, I do have some idea. I think this is where the change should go : https://github.com/keptn/keptn/blob/26b57f50cd53578ddda51e1c2fb03e4163ba767b/cli/cmd/upgrade.go#L61
I'll have to add a new flag and check if the skip-upgrade-check is set, if it is set I skip the above check else I continue as it is.
Is this the requirement?

Thanks for assigning, I do have some idea. I think this is where the change should go : https://github.com/keptn/keptn/blob/26b57f50cd53578ddda51e1c2fb03e4163ba767b/cli/cmd/upgrade.go#L61
I'll have to add a new flag and check if the skip-upgrade-check is set, if it is set I skip the above check else I continue as it is.
Is this the requirement?

Yes, taht's the right file, and within the doUpgradePreRunCheck function we have a call for checking if the release is compatible:
https://github.com/keptn/keptn/blob/26b57f50cd53578ddda51e1c2fb03e4163ba767b/cli/cmd/upgrade.go#L87-L98

Yes, taht's the right file, and within the doUpgradePreRunCheck function we have a call for checking if the release is compatible:
https://github.com/keptn/keptn/blob/26b57f50cd53578ddda51e1c2fb03e4163ba767b/cli/cmd/upgrade.go#L87-L98

Do you want to skip the doUpgradePreRunCheck completely or just the isUpgradeCompatible check?

Do you want to skip the doUpgradePreRunCheck completely or just the isUpgradeCompatible check?

I think it should only be the check for isUpgradeCompatible, the other checks will still need to be performed.

I think it should only be the check for isUpgradeCompatible, the other checks will still need to be performed.

Got it, I'll raise a PR with the changes. About the testing I don't see any existing test file for the upgrade command can you guide me regarding that?

Got it, I'll raise a PR with the changes. About the testing I don't see any existing test file for the upgrade command can you guide me regarding that?

You can create a file called upgrade_test.go in the same folder.
You can use https://github.com/keptn/keptn/blob/master/cli/cmd/generate_docs_test.go as a template.
Make sure to use the --mock flag.

You can create a file called upgrade_test.go in the same folder.
You can use https://github.com/keptn/keptn/blob/master/cli/cmd/generate_docs_test.go as a template.
Make sure to use the --mock flag.

hi @christian-kreuzberger-dtx, I have raised a PR for this issue #3020 please take a look and let me know if any changes are required.

hi @christian-kreuzberger-dtx, I have raised a PR for this issue #3020 please take a look and let me know if any changes are required.

authors: christian-kreuzberger-dtx gdsoumya


text:
Fix the index of the position in expert

The current position starts at 1 due to the cumulative sum, it should be start from 0.

LGTM

authors: hieuza ksalama

repo: keras-team/keras-io


text:
Repeated statement.

At line 140 there is line to store the residual. Same line is repeating as the next line.

Fixed, thank you for the report!

authors: fchollet sreagm


text:
ImageDataGenerator fed to 3D CNN in keras

Hey everyone, I am new in keras and python I am trying to use 3D CNN with keras, I did the following code shown below and there is an error relates to the dimension of the input shape:
The error says expecting 5d input but found 4d!! I know that 3D CNN input shape should be 5d but how can I reshape the train_data to be 5d? please.
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten, Conv3D, MaxPooling3D
from keras.layers.advanced_activations import LeakyReLU
from keras.optimizers import SGD, RMSprop
from keras.utils import np_utils, generic_utils
from keras.losses import categorical_crossentropy
from keras.optimizers import Adam
import os
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import cv2
from sklearn.cross_validation import train_test_split
from sklearn import cross_validation
from sklearn import preprocessing

datagen = ImageDataGenerator()
train_data=datagen.flow_from_directory('C:\\Users\\AA\\Data\\Training', target_size=(80, 100), color_mode='grayscale', classes=None, class_mode='categorical', batch_size=32, interpolation='nearest')
test_data=datagen.flow_from_directory('C:\\Users\\AA\\Data\\Testing', target_size=(80, 100), color_mode='grayscale', classes=None, class_mode='categorical', batch_size=32, interpolation='nearest')

    ins = (100, 100, 16, 1)
    model = Sequential()
    model.add(Conv3D(32, kernel_size=(3, 3, 3), input_shape=ins, padding=0))
    model.add(Activation('relu'))
    model.add(Conv3D(32, kernel_size=(3, 3, 3), padding=0))
    model.add(Activation('softmax'))
    model.add(MaxPooling3D(pool_size=(3, 3, 3), padding=0))
    model.add(Dropout(0.25))

    model.add(Conv3D(64, kernel_size=(3, 3, 3), padding=0))
    model.add(Activation('relu'))
    model.add(Conv3D(64, kernel_size=(3, 3, 3), padding=0))
    model.add(Activation('softmax'))
    model.add(MaxPooling3D(pool_size=(3, 3, 3), padding=0))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(Dense(512, activation='sigmoid'))
    model.add(Dropout(0.5))
    model.add(Dense(8, activation='softmax'))

 model.compile(loss=categorical_crossentropy, optimizer=Adam(), metrics=['accuracy'])

   model.fit_generator(train_data,
        steps_per_epoch=2000,
        epochs=50,
        validation_data=test_data,
        validation_steps=800)

Why do you want to use a 3D convolutional network for images? You don't have a third dimension.

The third dimension will be the time-steps which is the number of frames across time its 16 in my case.

Ah ok, I see. ImageDataGenerator does not support that out of the box, you will have to write a custom generator for this.

Any suggestion about that please? As I am completely new in python and keras!

Sure :) Check out this post on how to write an example custom generator: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html
You'll want to generate batches of shape (batch_size, time_steps, rows, cols, colors). This might mean something like (32, 16, 80, 100, 3) for you.

Hi portlion, I am in the same situation. Any updates so far? thanks

you can use my tweaked version of ImageGenerator class that fits this kind of problems.
https://gist.github.com/Emadeldeen-24/758b886c5cfe50498f1f4b8878eb0e24

@Emadeldeen-24 Hi, I tried to use your tweaked version but it also gives the error in fit:

ValueError: Input to .fit() should have rank 4. Got array with shape: (160, 128, 128, 128, 1)

@MarcelBeining I updated the comment and the code file. Please check and try again. If there's still a problem, please post the part of code causing the issue.

@Emadeldeen-24 Oh I see, so your way is not suited to load in tiff stack (i.e. 3d images) files right?

@Emadeldeen-24 i used your imageDataGenerator with @portlion 's code.
Here it is:  https://pastebin.com/ak6vJ4eq
But it get this error:
ValueError: Error when checking input: expected conv3d_14_input to have shape (100, 100, 16, 1) but got array with shape (4, 80, 100, 3)

Point#1: If the target_size=(80, 100) then the ins variable should become ins = (4, 80, 100, 1) (where 4 is the same as the value of frames_per_step parameter.
Point#2: if ins = (4, 80, 100, 1) then 1 means that you are using grayscale images not RGB ones (otherwise the 1 should become 3) but if you are using grayscale images then you should add color_mode='grayscale' in the train_data generator.

Thank you very much @Emadeldeen-24

@Emadeldeen-24 Thanks a lot for your contribution. However, if I understand correctly, it seems that you are reading frames_per_step images assuming that all of the images are stored consecutively and that each instance has the same number of timesteps.
However, usually for timeseries prediction (video recognition), each sequence of frames (timesteps) is stored in a folder, and each sequence need not have the same length of other sequences. That is, it would be great if we have a generator that would read the frames of each subfolder under each class (and zero pads it if necessary) and return it as a training example.

Thank you @EssamEmad for your comment.
What you are saying is partly right. I'm not assuming that images are stored consecutively in one folder, because even if they are stored in subfolders inside each class, ImageDataGenerator class will read them all as if they are together and will loop through the images/frames inside each folder.
However, I got your point and the thing you want to do can be achieved by setting an option shuffle=False as follows:
train_data=datagen.flow_from_directory('path/to/data', target_size=(x, y), batch_size=32, frames_per_step=4, shuffle=False)
In this way it will iterate over the images/frames inside each folder in the same order as they are and will pass through them folder by folder (even if it will not iterate over folders in there same order).
Note that the images inside each folder is better to be a multiple of "frames_per_step" value, otherwise you can write a small script that either deletes the increment or add some default images in each folder.
Hope this helps.

@Emadeldeen-24  Thanks for your comment. This is a great workaround. However, I have dived into the code, and I understand that if shuffle=False then the dataset is going to be fed in one class at a time. That is all of the instances of the first class would be yielded followed by all of the instances of the second..etc. This is not ideal for training a neural network as it might only remember the last class. Do you have any workarounds that you can think of?

@EssamEmad With the snippet below you can create your own 5D tensor. I am posting it, hope it helps:
frames_folder = 'path/to/folder'
 X_data = []
 y_data = []
 list_of_sent = os.listdir(frames_folder)
 print (list_of_sent)
 class_num = 0
 time_steps = 0
 frames = []
 for i in list_of_sent:
     classes_folder = str(frames_folder + '/' + i) #path to each class
     print (classes_folder)
     list_of_frames = os.listdir(classes_folder)
     time_steps= 0
     frames = []
     for filename in  sorted(list_of_frames):
         if ( time_steps == 8 ):
             X_data.append(frames) #appending each tensor of 8 frames resized to 110,110
             y_data.append(class_num) #appending a class label to the set of 8 frames
             j = 0
             frames = []
         else:
             time_steps+=1
             filename = cv2.imread(vid + '/' + filename)
             filename = cv2.resize(filename,(110, 110),interpolation=cv2.INTER_AREA)
             frames.append(filename)

     class_num+=1
 X_data = np.array(X_data)
 y_data = np.array(y_data)

For the snippet above, the folder structure must be like that:
  data/
         class0/
             img001.jpg
             img002.jpg
             ...
         class1/
             img001.jpg
             img002.jpg
             ...

@EssamEmad I don't think it will do so, because the selection of folders is random and not in order.

@FotisDionisopoulos I have already written a script like that that does the job. However, I am trying to use the ImageDataLoader to use the data augmentation and preprocessing easily.
@Emadeldeen-24 I have forked the script here and added a function shuffle_buckets which divides theindex_array into buckets, each of frames_per_step size. Then it shuffles those buckets so that the instances in the dataset are shuffled, but the relative order of each sequence is preserved.
However, when I tried it out, the shuffling works fine, but the output of the generator is a bit absurd. I have added print statements around 1073 to see the names of the files read, and it seems that a lot of files are read multiple times and the order isn't preserved. So, I tried the original code with shuffle=False and the same problem is still happening. I find plenty of sequences of frames that are of the same repeated image. That is each training example in each batch is a sequence of the same image repeated. Do you have any clue what's going on?

@Emadeldeen-24 Thanks a lot for your contribution. However, if I understand correctly, it seems that you are reading frames_per_step images assuming that all of the images are stored consecutively and that each instance has the same number of timesteps.
However, usually for timeseries prediction (video recognition), each sequence of frames (timesteps) is stored in a folder, and each sequence need not have the same length of other sequences. That is, it would be great if we have a generator that would read the frames of each subfolder under each class (and zero pads it if necessary) and return it as a training example.

Hi,
I understand your idea and want to implement the same. Have you implemented the same in python?

@Emadeldeen-24 Thanks a lot for your contribution. However, if I understand correctly, it seems that you are reading frames_per_step images assuming that all of the images are stored consecutively and that each instance has the same number of timesteps.
However, usually for timeseries prediction (video recognition), each sequence of frames (timesteps) is stored in a folder, and each sequence need not have the same length of other sequences. That is, it would be great if we have a generator that would read the frames of each subfolder under each class (and zero pads it if necessary) and return it as a training example.

Hi,
I understand your idea and want to implement the same. Have you implemented the same in python?

No I haven't. I wrote my own data generator and used Keras's original ImageDataGenerator.random_transform()  to do the data augmentation in my generator. I was initially having a hard-time figuring out how to use it, but it eventually worked. I would love to see this working though. Good luck (Y)

@Emadeldeen-24 Thanks a lot for your contribution. However, if I understand correctly, it seems that you are reading frames_per_step images assuming that all of the images are stored consecutively and that each instance has the same number of timesteps.
However, usually for timeseries prediction (video recognition), each sequence of frames (timesteps) is stored in a folder, and each sequence need not have the same length of other sequences. That is, it would be great if we have a generator that would read the frames of each subfolder under each class (and zero pads it if necessary) and return it as a training example.

Hi,
I understand your idea and want to implement the same. Have you implemented the same in python?

No I haven't. I wrote my own data generator and used Keras's original ImageDataGenerator.random_transform() to do the data augmentation in my generator. I was initially having a hard-time figuring out how to use it, but it eventually worked. I would love to see this working though. Good luck (Y)

Dear Emad,
Thank you for your reply.
I am actually struggling with the implementation. I have a problem as follows:

I have basketball match videos for which I have timestamps corresponding to each video (these timestamps denote when a shot is attempted).
I have converted each sub-clip of a video given at particular time stamp into frames
All the frames are stored in different folders corresponding to each timestamp (let us say frames related to shot "layup failure" is in one folder, and frames related to shot "layup success" is in another folder)
Now I want to build a 3D ConvNet where I can take these frames as my input and can predict whether the shot will be a "success" or a "fail" using first few frames

If you understand the problem, can you suggest me some possible approach to how it can be done?
I am attaching the link to the dataset where these timestamps are shown for each video.
http://basketballattention.appspot.com/dataset_browser.html
Any inputs would be highly appreciated.
Thanks!

@Emadeldeen-24 Thanks a lot for your contribution. However, if I understand correctly, it seems that you are reading frames_per_step images assuming that all of the images are stored consecutively and that each instance has the same number of timesteps.
However, usually for timeseries prediction (video recognition), each sequence of frames (timesteps) is stored in a folder, and each sequence need not have the same length of other sequences. That is, it would be great if we have a generator that would read the frames of each subfolder under each class (and zero pads it if necessary) and return it as a training example.

Hi,
I understand your idea and want to implement the same. Have you implemented the same in python?

No I haven't. I wrote my own data generator and used Keras's original ImageDataGenerator.random_transform() to do the data augmentation in my generator. I was initially having a hard-time figuring out how to use it, but it eventually worked. I would love to see this working though. Good luck (Y)

Dear Emad,
Thank you for your reply.
I am actually struggling with the implementation. I have a problem as follows:

I have basketball match videos for which I have timestamps corresponding to each video (these timestamps denote when a shot is attempted).
I have converted each sub-clip of a video given at particular time stamp into frames
All the frames are stored in different folders corresponding to each timestamp (let us say frames related to shot "layup failure" is in one folder, and frames related to shot "layup success" is in another folder)
Now I want to build a 3D ConvNet where I can take these frames as my input and can predict whether the shot will be a "success" or a "fail" using first few frames

If you understand the problem, can you suggest me some possible approach to how it can be done?
I am attaching the link to the dataset where these timestamps are shown for each video.
http://basketballattention.appspot.com/dataset_browser.html
Any inputs would be highly appreciated.
Thanks!

So, the dataset's link doesn't seem to work. However, if I understand you correctly, you can just write your own generator like the one suggested above. Where the frames of each training instance should be stored in a folder. The order they are going to be read in with os.listdir() is lexicographical. So, you should make sure that their corresponding timestep order is the same as the lexicographical. (That is, their names should be "00.jpg" "01.jpg" "21.jpg" ..etc)
Let me know if you have any other questions.

Hi @Emadeldeen-24  I am trying to use your codes for my time series cnn model. My x_train shape is (10000,10,90,90,1) but getting that error message:
`ValueError                                Traceback (most recent call last)
 in ()
1 from tweaked_ImageGenerator_v2 import ImageDataGenerator
2 datagen = ImageDataGenerator()
----> 3 train_data=datagen.flow_from_directory(x_train, target_size=(90, 90), batch_size=5000, frames_per_step=10,shuffle=False)
1 frames
/content/drive/Colab Notebooks/tweaked_ImageGenerator_v2.py in init(self, directory, image_data_generator, target_size, color_mode, classes, class_mode, frames_per_step, batch_size, shuffle, seed, data_format, save_to_dir, save_prefix, save_format, follow_links)
993         if not classes:
994             classes = []
--> 995             for subdir in sorted(os.listdir(directory)):
996                 if os.path.isdir(os.path.join(directory, subdir)):
997                     classes.append(subdir)
ValueError: listdir: embedded null character in path`
Do you have any idead about that problem?
I am using each 10 frames for my model.
Thanks
Emre

@etatbak Looks like you have an issue in the directory having the data (very probably have unrecognized characters). Make sure that path name is all english letters.
If this didn't help, show how your data is organized.

@etatbak Looks like you have an issue in the directory having the data (very probably have unrecognized characters). Make sure that path name is all english letters.
If this didn't help, show how your data is organized.

I wrote the path like tihs, because I already split my x_train inside of script, I don't have train data as saved. What do you think?
train_data=datagen.flow_from_directory(x_train, target_size=(90, 90), batch_size=5000, frames_per_step=10,shuffle=False)

it should be a path
https://keras.io/preprocessing/image/

it should be a path
https://keras.io/preprocessing/image/

The data, which format it should be? I tried with .pkl but having a problem.
Thanks for your help.

@Emadeldeen-24 Thanks a lot for your contribution. I tried to read your code and I have some doubt. I would like that the ImageDadaGenrator to apply the same deformations for all images in each set (given by frames_per_step). Does tweaked_ImageGenerator do this?
Thanks in advance.

Not sure about the deformations you want to apply, but I only updated the script so that it would accept multiple frames at each step.

@MarcelBeining hey mate, I just customised the Keras' ImageDataGenerator for medical image analysis applications, i.e. 3D Dicom slices (10,32,32,32,1). It might help with your problem.
https://github.com/dhuy228/augmented-volumetric-image-generator/blob/master/generator.py

@Emadeldeen-24 have you also made a script for predict_generator?

@Emadeldeen-24  and @dhuy228 , I get an error when I try to run your scripts and access DataGenerator or flow_from_directory like functions.
It says 'name K is not defined'.
This K is for Keras backend at the beginning of your scripts. I tried but I'm not able to resolve this. Please could someone help me with this? Thanks!

@Emadeldeen-24 I tried using your code. But it gives an error as follows:
AttributeError Traceback (most recent call last)
in
33 train_generator,
34 steps_per_epoch=22, # 2000 images = batch_size * steps
---> 35 epochs=100,
36 # validation_data=validation_generator,
37 # validation_steps=10, # 1000 images = batch_size * steps
~\Anaconda3\envs\tensorflow_gpuenv\lib\site-packages\tensorflow_core\python\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
1295 shuffle=shuffle,
1296 initial_epoch=initial_epoch,
-> 1297 steps_name='steps_per_epoch')
1298
1299 def evaluate_generator(self,
~\Anaconda3\envs\tensorflow_gpuenv\lib\site-packages\tensorflow_core\python\keras\engine\training_generator.py in model_iteration(model, data, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch, mode, batch_size, steps_name, **kwargs)
142 batch_size=batch_size,
143 epochs=epochs - initial_epoch,
--> 144 shuffle=shuffle)
145
146 do_validation = validation_data is not None
~\Anaconda3\envs\tensorflow_gpuenv\lib\site-packages\tensorflow_core\python\keras\engine\training_generator.py in convert_to_generator_like(data, batch_size, steps_per_epoch, epochs, shuffle)
475
476 # Create generator from NumPy or EagerTensor Input.
--> 477 num_samples = int(nest.flatten(data)[0].shape[0])
478 if batch_size is None:
479 raise ValueError(
AttributeError: 'DirectoryIterator' object has no attribute 'shape'
My input_shape is (7,32,32,1). It's 'channels_first' and 'grayscale' image.
Please let me know if you have some idea about fixing this. Thanks!

Your input shape is (7,32,32,1) which means that your image is of size 32x32x1 and your steps_per_epoch should be 7 but you've set it to 22.

Hi @ajeenkkya . Thanks for your comment. Sorry, I probably confused the readers by just mentioning the input shape. My image is 32x32x7 tiff stack and it is a grayscale image. I have around 108000 images and have set the batch size to 5000. This is why I had set steps_per_epoch to 22.
The input_shape that I set in Keras Sequential model code for the first convolution is (32,32,7,1) if I use 'channels_first' and set the same to (1,32,32,7) if I use 'channels_last'.
I also had 'frames_per_step = 7'. It still gives me the same error -> ''DirectoryIterator' object has no attribute 'shape''. Here's some of my code:
train_datagen = ImageDataGenerator(
rescale = 1./255,
rotation_range = 40,
width_shift_range = 0.2,
height_shift_range = 0.2,
horizontal_flip = True,
vertical_flip = True,
fill_mode = 'nearest',
data_format = 'channels_last')
test_datagen = ImageDataGenerator(rescale = 1./255)
Flow training images in batches of 5000 using train_datagen generator
train_generator = train_datagen.flow_from_directory(
train_dir,  # This is the source directory for training images
target_size=(32, 32),
batch_size=5000,
frames_per_step = 7,
class_mode='categorical',
color_mode = 'grayscale')
Flow validation images in batches of 2000 using test_datagen generator
validation_generator = test_datagen.flow_from_directory(
validation_dir,
target_size=(32, 32),
batch_size=2000,
frames_per_step = 7,
class_mode='categorical',
color_mode = 'grayscale')
history = model.fit_generator(
train_generator,
steps_per_epoch=22,  # 108000 images = batch_size * steps
epochs=100,
validation_data=validation_generator,
validation_steps=10,  # 19588 images = batch_size * steps
)
I'm not sure if this code is fine with a 'tif' stack or it expects an image sequence . Please let me know if you have some idea regarding the same. Thank you once again!

Hi @Emadeldeen-24, thanks for your code.
I have my data in folders, i.e:
TRAIN:
FOLDER1
FOLDER2
etc.
Each one of this folders contains 10 images named by:
name_001.jpg
name_002.jpg
name_003.jpg.
...
I'm using next code:
datagen = ImageDataGenerator()
train_data=datagen.flow_from_directory(train_dir, target_size=(128, 128), batch_size=64, frames_per_step=10, shuffle=False)
But, the result of train_data.filenames is random ( name_008.jpg  name_003.jpg  name_006.jpg...) . Id est, the images are not sorted. The code read the images into frames but the images are not in order. If we consider the option of serializing images, it doesn't make any sense that they are random. As I say, the shuffle parameter is false so I don't understand how I can fix it. Could you guide me?
Thanks in advance.

Hi @ajeenkkya . Thanks for your comment. Sorry, I probably confused the readers by just mentioning the input shape. My image is 32x32x7 tiff stack and it is a grayscale image. I have around 108000 images and have set the batch size to 5000. This is why I had set steps_per_epoch to 22.
The input_shape that I set in Keras Sequential model code for the first convolution is (32,32,7,1) if I use 'channels_first' and set the same to (1,32,32,7) if I use 'channels_last'.
I also had 'frames_per_step = 7'. It still gives me the same error -> ''DirectoryIterator' object has no attribute 'shape''. Here's some of my code:
train_datagen = ImageDataGenerator(
rescale = 1./255,
rotation_range = 40,
width_shift_range = 0.2,
height_shift_range = 0.2,
horizontal_flip = True,
vertical_flip = True,
fill_mode = 'nearest',
data_format = 'channels_last')
test_datagen = ImageDataGenerator(rescale = 1./255)
train_generator = train_datagen.flow_from_directory(
train_dir, # This is the source directory for training images
target_size=(32, 32),
batch_size=5000,
frames_per_step = 7,
class_mode='categorical',
color_mode = 'grayscale')
validation_generator = test_datagen.flow_from_directory(
validation_dir,
target_size=(32, 32),
batch_size=2000,
frames_per_step = 7,
class_mode='categorical',
color_mode = 'grayscale')
history = model.fit_generator(
train_generator,
steps_per_epoch=22, # 108000 images = batch_size * steps
epochs=100,
validation_data=validation_generator,
validation_steps=10, # 19588 images = batch_size * steps
)
I'm not sure if this code is fine with a 'tif' stack (all 7 images in a single tif file) or it expects an image sequence (a separate tif file for each of the 7 images) . Please let me know if you have some idea regarding the same. Thank you once again!

I think we have the same problem. Did you solve it already, the problem is the ImageDataGenerator just accept (x,y)  and when using the .fit_generator input image shape read as 2D rather than 3D.

Hi @redblast0308 . I couldn't fix the problem. However, an alternative way of doing it is loading the entire training and/or test data into numpy arrays before hand. After that use datagen.flow() function instead of datagen.flowfromdirectory() while creating training_generator and validation_generator. Since I had a grayscale image, I used my 7 slices/frames as 7 different channels. So I created my numpy arrays for train and test data with the following shapes:

train_data.shape = (no. of images for training, 32, 32, 7)
test_data.shape = (no. of images for testing, 32, 32, 7)

My code looked like this:
train_datagen = ImageDataGenerator(
rescale = 1./255,
rotation_range = 40,
width_shift_range = 0.2,
height_shift_range = 0.2,
horizontal_flip = True,
vertical_flip = True,
fill_mode = 'nearest',
data_format = 'channels_last')
test_datagen = ImageDataGenerator(rescale = 1./255)
Flow training images in batches of 400 using train_datagen generator
train_generator = train_datagen.flow(
train_data,
y_train,
batch_size=400)
validation_generator = test_datagen.flow(
test_data,
y_test,
batch_size=100)
fits the model on batches with real-time data augmentation:
history = model.fit_generator(
train_generator,
steps_per_epoch= math.ceil(len(train_data) / 400),
epochs=100,
validation_data = validation_generator,
validation_steps = math.ceil(len(test_data) / 100),
verbose = 2)
The .flow() function uses NumpyArrayIterator() which expects a maximum of 4 channels. However, the code still works after popping out a warning for 7 channels. And it seems to me that all my 7 channels are being used. I don't think this is a perfect solution for our problem, but this is one way to make it run or get started with. Hope it helps you!

Hello Sir, in my case, .tiff file was already stacked images.
On Thu, Apr 9, 2020 at 11:14 AM Suraj Deepak Khochare <
notifications@github.com> wrote:

Hi @redblast0308 https://github.com/redblast0308 . I couldn't fix the
problem. However, an alternative way of doing it is loading the entire
training and/or test data into numpy arrays before hand. After that use
datagen.flow() function instead of datagen.flowfromdirectory() while
creating training_generator and validation_generator. Since I had a
grayscale image, I used my 7 slices/frames as 7 different channels. So I
created my numpy arrays for train and test data with the following shapes:

train_data.shape = (no. of images for training, 32, 32, 7)
test_data.shape = (no. of images for testing, 32, 32, 7)

My code looked like this:
train_datagen = ImageDataGenerator(
rescale = 1./255,
rotation_range = 40,
width_shift_range = 0.2,
height_shift_range = 0.2,
horizontal_flip = True,
vertical_flip = True,
fill_mode = 'nearest',
data_format = 'channels_last')
test_datagen = ImageDataGenerator(rescale = 1./255)
Flow training images in batches of 400 using train_datagen generator
train_generator = train_datagen.flow(
train_data,
y_train,
batch_size=400)
validation_generator = test_datagen.flow(
test_data,
y_test,
batch_size=100)
fits the model on batches with real-time data augmentation:
history = model.fit_generator(
train_generator,
steps_per_epoch= math.ceil(len(train_data) / 400),
epochs=100,
validation_data = validation_generator,
validation_steps = math.ceil(len(test_data) / 100),
verbose = 2)
The .flow() function uses NumpyArrayIterator() which expects a maximum of
4 channels. However, the code still works after popping out a warning for 7
channels. And it seems to me that all my 7 channels are being used. I don't
think this is a perfect solution for our problem, but this is one way to
make it run or get started with. Hope it helps you!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10150#issuecomment-611304248,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW6CNI74ZLMYUIHPQ4TRLU4TDANCNFSM4E7CTQ3A
.

Hello Sir, in my case, .tiff file was already stacked images.
…
On Thu, Apr 9, 2020 at 11:14 AM Suraj Deepak Khochare < @.***> wrote: Hi @redblast0308 https://github.com/redblast0308 . I couldn't fix the problem. However, an alternative way of doing it is loading the entire training and/or test data into numpy arrays before hand. After that use datagen.flow() function instead of datagen.flowfromdirectory() while creating training_generator and validation_generator. Since I had a grayscale image, I used my 7 slices/frames as 7 different channels. So I created my numpy arrays for train and test data with the following shapes: 1. train_data.shape = (no. of images for training, 32, 32, 7) 2. test_data.shape = (no. of images for testing, 32, 32, 7) My code looked like this: train_datagen = ImageDataGenerator( rescale = 1./255, rotation_range = 40, width_shift_range = 0.2, height_shift_range = 0.2, horizontal_flip = True, vertical_flip = True, fill_mode = 'nearest', data_format = 'channels_last') test_datagen = ImageDataGenerator(rescale = 1./255) Flow training images in batches of 400 using train_datagen generator train_generator = train_datagen.flow( train_data, y_train, batch_size=400) validation_generator = test_datagen.flow( test_data, y_test, batch_size=100) fits the model on batches with real-time data augmentation: history = model.fit_generator( train_generator, steps_per_epoch= math.ceil(len(train_data) / 400), epochs=100, validation_data = validation_generator, validation_steps = math.ceil(len(test_data) / 100), verbose = 2) The .flow() function uses NumpyArrayIterator() which expects a maximum of 4 channels. However, the code still works after popping out a warning for 7 channels. And it seems to me that all my 7 channels are being used. I don't think this is a perfect solution for our problem, but this is one way to make it run or get started with. Hope it helps you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW6CNI74ZLMYUIHPQ4TRLU4TDANCNFSM4E7CTQ3A .

Hi! I think the best way forward is loading all the data into a numpy array and reshaping it according to your needs.

@FotisDionisopoulos I have already written a script like that that does the job. However, I am trying to use the ImageDataLoader to use the data augmentation and preprocessing easily.
@Emadeldeen-24 I have forked the script here and added a function shuffle_buckets which divides theindex_array into buckets, each of frames_per_step size. Then it shuffles those buckets so that the instances in the dataset are shuffled, but the relative order of each sequence is preserved.
However, when I tried it out, the shuffling works fine, but the output of the generator is a bit absurd. I have added print statements around 1073 to see the names of the files read, and it seems that a lot of files are read multiple times and the order isn't preserved. So, I tried the original code with shuffle=False and the same problem is still happening. I find plenty of sequences of frames that are of the same repeated image. That is each training example in each batch is a sequence of the same image repeated. Do you have any clue what's going on?

Did you get it to work after this? I'm working on a project and could use the shuffle_buckets approach.

Yes, I got it already. by using the normal model.fit in keras what I did is
I stacked the image into 3D then call it by folder as classes
On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal notifications@github.com
wrote:

@FotisDionisopoulos https://github.com/FotisDionisopoulos I have
already written a script like that that does the job. However, I am trying
to use the ImageDataLoader to use the data augmentation and preprocessing
easily.
@Emadeldeen-24 https://github.com/Emadeldeen-24 I have forked the
script here
https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and
added a function shuffle_buckets which divides theindex_array into
buckets, each of frames_per_step size. Then it shuffles those buckets so
that the instances in the dataset are shuffled, but the relative order of
each sequence is preserved.
However, when I tried it out, the shuffling works fine, but the output of
the generator is a bit absurd. I have added print statements around 1073 to
see the names of the files read, and it seems that a lot of files are read
multiple times and the order isn't preserved. So, I tried the original code
with shuffle=False and the same problem is still happening. I find plenty
of sequences of frames that are of the same repeated image. That is each
training example in each batch is a sequence of the same image repeated. Do
you have any clue what's going on?
Did you get it to work after this? I'm working on a project and could use
the shuffle_buckets approach.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10150#issuecomment-651201218,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A
.

Yes, I got it already. by using the normal model.fit in keras what I did is I stacked the image into 3D then call it by folder as classes
…
On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal @.***> wrote: @FotisDionisopoulos https://github.com/FotisDionisopoulos I have already written a script like that that does the job. However, I am trying to use the ImageDataLoader to use the data augmentation and preprocessing easily. @Emadeldeen-24 https://github.com/Emadeldeen-24 I have forked the script here https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and added a function shuffle_buckets which divides theindex_array into buckets, each of frames_per_step size. Then it shuffles those buckets so that the instances in the dataset are shuffled, but the relative order of each sequence is preserved. However, when I tried it out, the shuffling works fine, but the output of the generator is a bit absurd. I have added print statements around 1073 to see the names of the files read, and it seems that a lot of files are read multiple times and the order isn't preserved. So, I tried the original code with shuffle=False and the same problem is still happening. I find plenty of sequences of frames that are of the same repeated image. That is each training example in each batch is a sequence of the same image repeated. Do you have any clue what's going on? Did you get it to work after this? I'm working on a project and could use the shuffle_buckets approach. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A .

Did you stack it in the channels dimension? What do you mean call it by folder as classes?
I'm working on a classification task on CT scan data, with each patient data stored in .nii files. I have separated and stored all individual slices in png format in their respective folders corresponding to the class they belong to (healthy and infected). For my task I need to input a particular number of slices, feed each one into a CNN , and combine their features to feed in a Dense layer. How do I take input for such a model if let's say I want to pass 4 individual slices as input?

What I did was I stacked the .png to 3D .tiff file in my case those became
128x128x20 before feeding to the 3D CNN. 20 here is the depth/channel. I
converted the . png into arrays then I used volume io to stacked it in 3D.
On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal notifications@github.com
wrote:

Yes, I got it already. by using the normal model.fit in keras what I did
is I stacked the image into 3D then call it by folder as classes
… <#m_-228565096952996164_>
On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal @.***> wrote:
@FotisDionisopoulos https://github.com/FotisDionisopoulos
https://github.com/FotisDionisopoulos I have already written a script
like that that does the job. However, I am trying to use the
ImageDataLoader to use the data augmentation and preprocessing easily.
@Emadeldeen-24 https://github.com/Emadeldeen-24
https://github.com/Emadeldeen-24 I have forked the script here
https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and
added a function shuffle_buckets which divides theindex_array into buckets,
each of frames_per_step size. Then it shuffles those buckets so that the
instances in the dataset are shuffled, but the relative order of each
sequence is preserved. However, when I tried it out, the shuffling works
fine, but the output of the generator is a bit absurd. I have added print
statements around 1073 to see the names of the files read, and it seems
that a lot of files are read multiple times and the order isn't preserved.
So, I tried the original code with shuffle=False and the same problem is
still happening. I find plenty of sequences of frames that are of the same
repeated image. That is each training example in each batch is a sequence
of the same image repeated. Do you have any clue what's going on? Did you
get it to work after this? I'm working on a project and could use the
shuffle_buckets approach. — You are receiving this because you were
mentioned. Reply to this email directly, view it on GitHub <#10150
(comment)
https://github.com/keras-team/keras/issues/10150#issuecomment-651201218>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A
.
Did you stack it in the channels dimension? What do you mean call it by
folder as classes?
I'm working on a classification task on CT scan data, with each patient
data stored in .nii files. I have separated and stored all individual
slices in png format in their respective folders corresponding to the class
they belong to (healthy and infected). For my task I need to input a
particular number of slices, feed each one into a CNN , and combine their
features to feed in a Dense layer. How do I take input for such a model if
let's say I want to pass 4 individual slices as input?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10150#issuecomment-651804164,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A
.

By doing that. I directly used model.fit which can accept the necessary
dimensions as input in 3D CNN without using image generator.
On Tue, Jun 30, 2020, 10:23 PM rolando pula pularolando@gmail.com wrote:

What I did was I stacked the .png to 3D .tiff file in my case those became
128x128x20 before feeding to the 3D CNN. 20 here is the depth/channel. I
converted the . png into arrays then I used volume io to stacked it in 3D.
On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal notifications@github.com
wrote:

Yes, I got it already. by using the normal model.fit in keras what I did
is I stacked the image into 3D then call it by folder as classes
… <#m_-621567969079526466_m_-228565096952996164_>
On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal @.***> wrote:
@FotisDionisopoulos https://github.com/FotisDionisopoulos
https://github.com/FotisDionisopoulos I have already written a script
like that that does the job. However, I am trying to use the
ImageDataLoader to use the data augmentation and preprocessing easily.
@Emadeldeen-24 https://github.com/Emadeldeen-24
https://github.com/Emadeldeen-24 I have forked the script here
https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and
added a function shuffle_buckets which divides theindex_array into buckets,
each of frames_per_step size. Then it shuffles those buckets so that the
instances in the dataset are shuffled, but the relative order of each
sequence is preserved. However, when I tried it out, the shuffling works
fine, but the output of the generator is a bit absurd. I have added print
statements around 1073 to see the names of the files read, and it seems
that a lot of files are read multiple times and the order isn't preserved.
So, I tried the original code with shuffle=False and the same problem is
still happening. I find plenty of sequences of frames that are of the same
repeated image. That is each training example in each batch is a sequence
of the same image repeated. Do you have any clue what's going on? Did you
get it to work after this? I'm working on a project and could use the
shuffle_buckets approach. — You are receiving this because you were
mentioned. Reply to this email directly, view it on GitHub <#10150
(comment)
https://github.com/keras-team/keras/issues/10150#issuecomment-651201218>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A
.
Did you stack it in the channels dimension? What do you mean call it by
folder as classes?
I'm working on a classification task on CT scan data, with each patient
data stored in .nii files. I have separated and stored all individual
slices in png format in their respective folders corresponding to the class
they belong to (healthy and infected). For my task I need to input a
particular number of slices, feed each one into a CNN , and combine their
features to feed in a Dense layer. How do I take input for such a model if
let's say I want to pass 4 individual slices as input?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10150#issuecomment-651804164,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A
.

You can still used the data preprocessing by normalizing the values. That
is only preprocessing you can do without using the image generator
On Tue, Jun 30, 2020, 10:26 PM rolando pula pularolando@gmail.com wrote:

By doing that. I directly used model.fit which can accept the necessary
dimensions as input in 3D CNN without using image generator.
On Tue, Jun 30, 2020, 10:23 PM rolando pula pularolando@gmail.com wrote:

What I did was I stacked the .png to 3D .tiff file in my case those
became 128x128x20 before feeding to the 3D CNN. 20 here is the
depth/channel. I converted the . png into arrays then I used volume io to
stacked it in 3D.
On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal notifications@github.com
wrote:

Yes, I got it already. by using the normal model.fit in keras what I did
is I stacked the image into 3D then call it by folder as classes
… <#m_5543591580139675636_m_-621567969079526466_m_-228565096952996164_>
On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal @.***> wrote:
@FotisDionisopoulos https://github.com/FotisDionisopoulos
https://github.com/FotisDionisopoulos I have already written a script
like that that does the job. However, I am trying to use the
ImageDataLoader to use the data augmentation and preprocessing easily.
@Emadeldeen-24 https://github.com/Emadeldeen-24
https://github.com/Emadeldeen-24 I have forked the script here
https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and
added a function shuffle_buckets which divides theindex_array into buckets,
each of frames_per_step size. Then it shuffles those buckets so that the
instances in the dataset are shuffled, but the relative order of each
sequence is preserved. However, when I tried it out, the shuffling works
fine, but the output of the generator is a bit absurd. I have added print
statements around 1073 to see the names of the files read, and it seems
that a lot of files are read multiple times and the order isn't preserved.
So, I tried the original code with shuffle=False and the same problem is
still happening. I find plenty of sequences of frames that are of the same
repeated image. That is each training example in each batch is a sequence
of the same image repeated. Do you have any clue what's going on? Did you
get it to work after this? I'm working on a project and could use the
shuffle_buckets approach. — You are receiving this because you were
mentioned. Reply to this email directly, view it on GitHub <#10150
(comment)
https://github.com/keras-team/keras/issues/10150#issuecomment-651201218>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A
.
Did you stack it in the channels dimension? What do you mean call it by
folder as classes?
I'm working on a classification task on CT scan data, with each patient
data stored in .nii files. I have separated and stored all individual
slices in png format in their respective folders corresponding to the class
they belong to (healthy and infected). For my task I need to input a
particular number of slices, feed each one into a CNN , and combine their
features to feed in a Dense layer. How do I take input for such a model if
let's say I want to pass 4 individual slices as input?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10150#issuecomment-651804164,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A
.

Okay got it. Thanks for your help!
Will try to work my way around image generator since I need augmentation too.

If you image generator is not working. I'll try to share my code with you.
On Wed, Jul 1, 2020, 12:48 AM Utkarsh Agarwal notifications@github.com
wrote:

Okay got it. Thanks for your help!
Will try to work my way around image generator since I need augmentation
too.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10150#issuecomment-651915348,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW22Z6B7TFUPDILW2MDRZIJNRANCNFSM4E7CTQ3A
.

@EssamEmad With the snippet below you can create your own 5D tensor. I am posting it, hope it helps:
frames_folder = 'path/to/folder'
 X_data = []
 y_data = []
 list_of_sent = os.listdir(frames_folder)
 print (list_of_sent)
 class_num = 0
 time_steps = 0
 frames = []
 for i in list_of_sent:
     classes_folder = str(frames_folder + '/' + i) #path to each class
     print (classes_folder)
     list_of_frames = os.listdir(classes_folder)
     time_steps= 0
     frames = []
     for filename in  sorted(list_of_frames):
         if ( time_steps == 8 ):
             X_data.append(frames) #appending each tensor of 8 frames resized to 110,110
             y_data.append(class_num) #appending a class label to the set of 8 frames
             j = 0
             frames = []
         else:
             time_steps+=1
             filename = cv2.imread(vid + '/' + filename)
             filename = cv2.resize(filename,(110, 110),interpolation=cv2.INTER_AREA)
             frames.append(filename)

     class_num+=1
 X_data = np.array(X_data)
 y_data = np.array(y_data)

For the snippet above, the folder structure must be like that:
  data/
         class0/
             img001.jpg
             img002.jpg
             ...
         class1/
             img001.jpg
             img002.jpg
             ...

What does the vid variable refer to on line 23?

You can still used the data preprocessing by normalizing the values. That is only preprocessing you can do without using the image generator
…
On Tue, Jun 30, 2020, 10:26 PM rolando pula @.> wrote: By doing that. I directly used model.fit which can accept the necessary dimensions as input in 3D CNN without using image generator. On Tue, Jun 30, 2020, 10:23 PM rolando pula @.> wrote: > What I did was I stacked the .png to 3D .tiff file in my case those > became 128x128x20 before feeding to the 3D CNN. 20 here is the > depth/channel. I converted the . png into arrays then I used volume io to > stacked it in 3D. > > On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal @.> > wrote: > >> Yes, I got it already. by using the normal model.fit in keras what I did >> is I stacked the image into 3D then call it by folder as classes >> … <#m_5543591580139675636_m_-621567969079526466_m_-228565096952996164_> >> On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal @.> wrote: >> @FotisDionisopoulos https://github.com/FotisDionisopoulos >> https://github.com/FotisDionisopoulos I have already written a script >> like that that does the job. However, I am trying to use the >> ImageDataLoader to use the data augmentation and preprocessing easily. >> @Emadeldeen-24 https://github.com/Emadeldeen-24 >> https://github.com/Emadeldeen-24 I have forked the script here >> https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and >> added a function shuffle_buckets which divides theindex_array into buckets, >> each of frames_per_step size. Then it shuffles those buckets so that the >> instances in the dataset are shuffled, but the relative order of each >> sequence is preserved. However, when I tried it out, the shuffling works >> fine, but the output of the generator is a bit absurd. I have added print >> statements around 1073 to see the names of the files read, and it seems >> that a lot of files are read multiple times and the order isn't preserved. >> So, I tried the original code with shuffle=False and the same problem is >> still happening. I find plenty of sequences of frames that are of the same >> repeated image. That is each training example in each batch is a sequence >> of the same image repeated. Do you have any clue what's going on? Did you >> get it to work after this? I'm working on a project and could use the >> shuffle_buckets approach. — You are receiving this because you were >> mentioned. Reply to this email directly, view it on GitHub <#10150 >> (comment) >> <#10150 (comment)>>, >> or unsubscribe >> https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A >> . >> >> Did you stack it in the channels dimension? What do you mean call it by >> folder as classes? >> I'm working on a classification task on CT scan data, with each patient >> data stored in .nii files. I have separated and stored all individual >> slices in png format in their respective folders corresponding to the class >> they belong to (healthy and infected). For my task I need to input a >> particular number of slices, feed each one into a CNN , and combine their >> features to feed in a Dense layer. How do I take input for such a model if >> let's say I want to pass 4 individual slices as input? >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#10150 (comment)>, >> or unsubscribe >> https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A >> . >> >

@redblast0308 Hi, I am working on the mri data where I stacked 60 slices in the z-axis as a depth, (width, height, depth- 96x96x60) and now I have to use image data generator to augment the data before fitting to the model. Could you please help me out how did you do that?

Okay got it. Thanks for your help!
Will try to work my way around image generator since I need augmentation too.

Hi @utkarsh1205, I am working on my

What I did was I stacked the .png to 3D .tiff file in my case those became 128x128x20 before feeding to the 3D CNN. 20 here is the depth/channel. I converted the . png into arrays then I used volume io to stacked it in 3D. On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal notifications@github.com wrote:
…
Yes, I got it already. by using the normal model.fit in keras what I did is I stacked the image into 3D then call it by folder as classes … <#m_-228565096952996164_> On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal @.***> wrote: @FotisDionisopoulos https://github.com/FotisDionisopoulos https://github.com/FotisDionisopoulos I have already written a script like that that does the job. However, I am trying to use the ImageDataLoader to use the data augmentation and preprocessing easily. @Emadeldeen-24 https://github.com/Emadeldeen-24 https://github.com/Emadeldeen-24 I have forked the script here https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and added a function shuffle_buckets which divides theindex_array into buckets, each of frames_per_step size. Then it shuffles those buckets so that the instances in the dataset are shuffled, but the relative order of each sequence is preserved. However, when I tried it out, the shuffling works fine, but the output of the generator is a bit absurd. I have added print statements around 1073 to see the names of the files read, and it seems that a lot of files are read multiple times and the order isn't preserved. So, I tried the original code with shuffle=False and the same problem is still happening. I find plenty of sequences of frames that are of the same repeated image. That is each training example in each batch is a sequence of the same image repeated. Do you have any clue what's going on? Did you get it to work after this? I'm working on a project and could use the shuffle_buckets approach. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 (comment) <#10150 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A . Did you stack it in the channels dimension? What do you mean call it by folder as classes? I'm working on a classification task on CT scan data, with each patient data stored in .nii files. I have separated and stored all individual slices in png format in their respective folders corresponding to the class they belong to (healthy and infected). For my task I need to input a particular number of slices, feed each one into a CNN , and combine their features to feed in a Dense layer. How do I take input for such a model if let's say I want to pass 4 individual slices as input? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A .

Hi @redblast0308, I am in the same situation. I stacked slices on z-axis and created new dimension (96, 96, 60), converted into array, did normalize too and fed to the 3D model but I haven't received promising accuracy at the end.
Next, I used keras imagedatagenerator for augmentation but here it won't take more than 4 channels. Could you please share how did you do augmentation on your data with (128, 128, 20)? Any help would be highly appreciated. Thanks!

The only augmentation/transformation I did is before stacking the image
into volumetric. What I did was used Markov transition field for signal to
image. You can do augmentation before stacking image because in my case I
did not used imagedatagenerator. But in the github and stackoverflow forums
they used custom imagedatagenerator. I got 98% percent accuracy in my ase
by tuning the model. Tried first to tuned it using grid search.
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
Virus-free.
www.avast.com
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Thu, Dec 31, 2020 at 1:03 AM dilipv09 notifications@github.com wrote:

Okay got it. Thanks for your help!
Will try to work my way around image generator since I need augmentation
too.
Hi @utkarsh1205 https://github.com/utkarsh1205, I am working on my
What I did was I stacked the .png to 3D .tiff file in my case those became
128x128x20 before feeding to the 3D CNN. 20 here is the depth/channel. I
converted the . png into arrays then I used volume io to stacked it in 3D.
On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal notifications@github.com
wrote:
… <#m_8283731420964725201_>
Yes, I got it already. by using the normal model.fit in keras what I did
is I stacked the image into 3D then call it by folder as classes …
<#m_-228565096952996164_> On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal
@.***> wrote: @FotisDionisopoulos
https://github.com/FotisDionisopoulos
https://github.com/FotisDionisopoulos
https://github.com/FotisDionisopoulos I have already written a script
like that that does the job. However, I am trying to use the
ImageDataLoader to use the data augmentation and preprocessing easily.
@Emadeldeen-24 https://github.com/Emadeldeen-24
https://github.com/Emadeldeen-24 I have forked the script here
https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and
added a function shuffle_buckets which divides theindex_array into buckets,
each of frames_per_step size. Then it shuffles those buckets so that the
instances in the dataset are shuffled, but the relative order of each
sequence is preserved. However, when I tried it out, the shuffling works
fine, but the output of the generator is a bit absurd. I have added print
statements around 1073 to see the names of the files read, and it seems
that a lot of files are read multiple times and the order isn't preserved.
So, I tried the original code with shuffle=False and the same problem is
still happening. I find plenty of sequences of frames that are of the same
repeated image. That is each training example in each batch is a sequence
of the same image repeated. Do you have any clue what's going on? Did you
get it to work after this? I'm working on a project and could use the
shuffle_buckets approach. — You are receiving this because you were
mentioned. Reply to this email directly, view it on GitHub <#10150
https://github.com/keras-team/keras/issues/10150 (comment) <#10150
(comment)
https://github.com/keras-team/keras/issues/10150#issuecomment-651201218>>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A
. Did you stack it in the channels dimension? What do you mean call it by
folder as classes? I'm working on a classification task on CT scan data,
with each patient data stored in .nii files. I have separated and stored
all individual slices in png format in their respective folders
corresponding to the class they belong to (healthy and infected). For my
task I need to input a particular number of slices, feed each one into a
CNN , and combine their features to feed in a Dense layer. How do I take
input for such a model if let's say I want to pass 4 individual slices as
input? — You are receiving this because you were mentioned. Reply to this
email directly, view it on GitHub <#10150 (comment)
https://github.com/keras-team/keras/issues/10150#issuecomment-651804164>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A
.
Hi @redblast0308 https://github.com/redblast0308, I am in the same
situation. I stacked slices on z-axis and created new dimension (96, 96,
60), converted into array, did normalize too and fed to the 3D model but I
haven't received promising accuracy at the end.
Next, I used keras imagedatagenerator for augmentation but here it won't
take more than 4 channels. Could you please share how did you do
augmentation on your data with (128, 128, 20)? Any help would be highly
appreciated. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10150#issuecomment-752691306,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW7235OTGMSH6SUUXCTSXNMMRANCNFSM4E7CTQ3A
.

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
Virus-free.
www.avast.com
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

The only augmentation/transformation I did is before stacking the image
into volumetric. What I did was used Markov transition field for signal to
image. You can do augmentation before stacking image because in my case I
did not used imagedatagenerator. But in the github and stackoverflow forums
they used custom imagedatagenerator. I got 98% percent accuracy in my ase
by tuning the model. Tried first to tuned it using grid search.
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
Virus-free.
www.avast.com
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Thu, Dec 31, 2020 at 1:03 AM dilipv09 notifications@github.com wrote:

Okay got it. Thanks for your help!
Will try to work my way around image generator since I need augmentation
too.
Hi @utkarsh1205 https://github.com/utkarsh1205, I am working on my
What I did was I stacked the .png to 3D .tiff file in my case those became
128x128x20 before feeding to the 3D CNN. 20 here is the depth/channel. I
converted the . png into arrays then I used volume io to stacked it in 3D.
On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal notifications@github.com
wrote:
… <#m_8283731420964725201_>
Yes, I got it already. by using the normal model.fit in keras what I did
is I stacked the image into 3D then call it by folder as classes …
<#m_-228565096952996164_> On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal
@.***> wrote: @FotisDionisopoulos
https://github.com/FotisDionisopoulos
https://github.com/FotisDionisopoulos
https://github.com/FotisDionisopoulos I have already written a script
like that that does the job. However, I am trying to use the
ImageDataLoader to use the data augmentation and preprocessing easily.
@Emadeldeen-24 https://github.com/Emadeldeen-24
https://github.com/Emadeldeen-24 I have forked the script here
https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and
added a function shuffle_buckets which divides theindex_array into buckets,
each of frames_per_step size. Then it shuffles those buckets so that the
instances in the dataset are shuffled, but the relative order of each
sequence is preserved. However, when I tried it out, the shuffling works
fine, but the output of the generator is a bit absurd. I have added print
statements around 1073 to see the names of the files read, and it seems
that a lot of files are read multiple times and the order isn't preserved.
So, I tried the original code with shuffle=False and the same problem is
still happening. I find plenty of sequences of frames that are of the same
repeated image. That is each training example in each batch is a sequence
of the same image repeated. Do you have any clue what's going on? Did you
get it to work after this? I'm working on a project and could use the
shuffle_buckets approach. — You are receiving this because you were
mentioned. Reply to this email directly, view it on GitHub <#10150
https://github.com/keras-team/keras/issues/10150 (comment) <#10150
(comment)
https://github.com/keras-team/keras/issues/10150#issuecomment-651201218>>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A
. Did you stack it in the channels dimension? What do you mean call it by
folder as classes? I'm working on a classification task on CT scan data,
with each patient data stored in .nii files. I have separated and stored
all individual slices in png format in their respective folders
corresponding to the class they belong to (healthy and infected). For my
task I need to input a particular number of slices, feed each one into a
CNN , and combine their features to feed in a Dense layer. How do I take
input for such a model if let's say I want to pass 4 individual slices as
input? — You are receiving this because you were mentioned. Reply to this
email directly, view it on GitHub <#10150 (comment)
https://github.com/keras-team/keras/issues/10150#issuecomment-651804164>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A
.
Hi @redblast0308 https://github.com/redblast0308, I am in the same
situation. I stacked slices on z-axis and created new dimension (96, 96,
60), converted into array, did normalize too and fed to the 3D model but I
haven't received promising accuracy at the end.
Next, I used keras imagedatagenerator for augmentation but here it won't
take more than 4 channels. Could you please share how did you do
augmentation on your data with (128, 128, 20)? Any help would be highly
appreciated. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10150#issuecomment-752691306,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ4NPW7235OTGMSH6SUUXCTSXNMMRANCNFSM4E7CTQ3A
.

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
Virus-free.
www.avast.com
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

The only augmentation/transformation I did is before stacking the image into volumetric. What I did was used Markov transition field for signal to image. You can do augmentation before stacking image because in my case I did not used imagedatagenerator. But in the github and stackoverflow forums they used custom imagedatagenerator. I got 98% percent accuracy in my ase by tuning the model. Tried first to tuned it using grid search. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Thu, Dec 31, 2020 at 1:03 AM dilipv09 @.> wrote: Okay got it. Thanks for your help! Will try to work my way around image generator since I need augmentation too. Hi @utkarsh1205 https://github.com/utkarsh1205, I am working on my What I did was I stacked the .png to 3D .tiff file in my case those became 128x128x20 before feeding to the 3D CNN. 20 here is the depth/channel. I converted the . png into arrays then I used volume io to stacked it in 3D. On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal @. wrote: … <#m_8283731420964725201_> Yes, I got it already. by using the normal model.fit in keras what I did is I stacked the image into 3D then call it by folder as classes … <#m_-228565096952996164_> On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal @.***> wrote: @FotisDionisopoulos https://github.com/FotisDionisopoulos https://github.com/FotisDionisopoulos https://github.com/FotisDionisopoulos I have already written a script like that that does the job. However, I am trying to use the ImageDataLoader to use the data augmentation and preprocessing easily. @Emadeldeen-24 https://github.com/Emadeldeen-24 https://github.com/Emadeldeen-24 I have forked the script here https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and added a function shuffle_buckets which divides theindex_array into buckets, each of frames_per_step size. Then it shuffles those buckets so that the instances in the dataset are shuffled, but the relative order of each sequence is preserved. However, when I tried it out, the shuffling works fine, but the output of the generator is a bit absurd. I have added print statements around 1073 to see the names of the files read, and it seems that a lot of files are read multiple times and the order isn't preserved. So, I tried the original code with shuffle=False and the same problem is still happening. I find plenty of sequences of frames that are of the same repeated image. That is each training example in each batch is a sequence of the same image repeated. Do you have any clue what's going on? Did you get it to work after this? I'm working on a project and could use the shuffle_buckets approach. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 <#10150> (comment) <#10150 (comment) <#10150 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A . Did you stack it in the channels dimension? What do you mean call it by folder as classes? I'm working on a classification task on CT scan data, with each patient data stored in .nii files. I have separated and stored all individual slices in png format in their respective folders corresponding to the class they belong to (healthy and infected). For my task I need to input a particular number of slices, feed each one into a CNN , and combine their features to feed in a Dense layer. How do I take input for such a model if let's say I want to pass 4 individual slices as input? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 (comment) <#10150 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A . Hi @redblast0308 https://github.com/redblast0308, I am in the same situation. I stacked slices on z-axis and created new dimension (96, 96, 60), converted into array, did normalize too and fed to the 3D model but I haven't received promising accuracy at the end. Next, I used keras imagedatagenerator for augmentation but here it won't take more than 4 channels. Could you please share how did you do augmentation on your data with (128, 128, 20)? Any help would be highly appreciated. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW7235OTGMSH6SUUXCTSXNMMRANCNFSM4E7CTQ3A .
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

@redblast0308 Thanks for sharing the information. This is helpful. Appreciated!

The only augmentation/transformation I did is before stacking the image into volumetric. What I did was used Markov transition field for signal to image. You can do augmentation before stacking image because in my case I did not used imagedatagenerator. But in the github and stackoverflow forums they used custom imagedatagenerator. I got 98% percent accuracy in my ase by tuning the model. Tried first to tuned it using grid search. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Thu, Dec 31, 2020 at 1:03 AM dilipv09 @.> wrote: Okay got it. Thanks for your help! Will try to work my way around image generator since I need augmentation too. Hi @utkarsh1205 https://github.com/utkarsh1205, I am working on my What I did was I stacked the .png to 3D .tiff file in my case those became 128x128x20 before feeding to the 3D CNN. 20 here is the depth/channel. I converted the . png into arrays then I used volume io to stacked it in 3D. On Tue, Jun 30, 2020, 9:51 PM Utkarsh Agarwal @. wrote: … <#m_8283731420964725201_> Yes, I got it already. by using the normal model.fit in keras what I did is I stacked the image into 3D then call it by folder as classes … <#m_-228565096952996164_> On Mon, Jun 29, 2020 at 11:41 PM Utkarsh Agarwal @.***> wrote: @FotisDionisopoulos https://github.com/FotisDionisopoulos https://github.com/FotisDionisopoulos https://github.com/FotisDionisopoulos I have already written a script like that that does the job. However, I am trying to use the ImageDataLoader to use the data augmentation and preprocessing easily. @Emadeldeen-24 https://github.com/Emadeldeen-24 https://github.com/Emadeldeen-24 I have forked the script here https://gist.github.com/EssamEmad/4a7ffc40db820a8d5bdae92dc0e780ff and added a function shuffle_buckets which divides theindex_array into buckets, each of frames_per_step size. Then it shuffles those buckets so that the instances in the dataset are shuffled, but the relative order of each sequence is preserved. However, when I tried it out, the shuffling works fine, but the output of the generator is a bit absurd. I have added print statements around 1073 to see the names of the files read, and it seems that a lot of files are read multiple times and the order isn't preserved. So, I tried the original code with shuffle=False and the same problem is still happening. I find plenty of sequences of frames that are of the same repeated image. That is each training example in each batch is a sequence of the same image repeated. Do you have any clue what's going on? Did you get it to work after this? I'm working on a project and could use the shuffle_buckets approach. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 <#10150> (comment) <#10150 (comment) <#10150 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW7PZCP7GN2D56GFQOTRZCY2PANCNFSM4E7CTQ3A . Did you stack it in the channels dimension? What do you mean call it by folder as classes? I'm working on a classification task on CT scan data, with each patient data stored in .nii files. I have separated and stored all individual slices in png format in their respective folders corresponding to the class they belong to (healthy and infected). For my task I need to input a particular number of slices, feed each one into a CNN , and combine their features to feed in a Dense layer. How do I take input for such a model if let's say I want to pass 4 individual slices as input? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 (comment) <#10150 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW5AKH3VYNNJJEP4MCTRZHUWVANCNFSM4E7CTQ3A . Hi @redblast0308 https://github.com/redblast0308, I am in the same situation. I stacked slices on z-axis and created new dimension (96, 96, 60), converted into array, did normalize too and fed to the 3D model but I haven't received promising accuracy at the end. Next, I used keras imagedatagenerator for augmentation but here it won't take more than 4 channels. Could you please share how did you do augmentation on your data with (128, 128, 20)? Any help would be highly appreciated. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10150 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ4NPW7235OTGMSH6SUUXCTSXNMMRANCNFSM4E7CTQ3A .
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

@redblast0308 Thanks for sharing the information. This is helpful. Appreciated!

You can use my tweaked version of ImageDataGenerator class that fits this kind of problems.
It's the same as the main Keras ImageDataGenerator but I added an option to take more than one image/frame on each iteration. This is by changing the parameter frames_per_step to specify the number of frames/images you want to include in each iteration.
So here's how to use it:
from tweaked_ImageGenerator_v2 import ImageDataGenerator
datagen = ImageDataGenerator()
train_data=datagen.flow_from_directory('path/to/data', target_size=(x, y), batch_size=32, frames_per_step=4)

@emadeldeen24 , thank you for the customer image generator, I want to have an output shape (number_of_batches, size_x, size_y, time_steps, 1). Is it possible to modify your source code and if yes, can you help in doing it?

in this file(tweaked_ImageGenerator_v2), use an Image(png, jpg) file, is there any way to make the .nii file read?
( not use PIL, use nibabel to read .nii file)

repo: keras-team/keras

license_source: bigquery


text:
Fit Generator different checkpoint metric results using same generator for training and validation

I'm wondering why I get different checkpoint metrics using the same generator for training and validation. I'm using keras version 2.0.9 with tensorflow 1.4.1. I know using the same generator does not make much sense, but I started doing this for debugging and got intrigued by this issue.
history = m.fit_generator(G, steps_per_epoch=len(params.trainImageList) / params.train_batch_size, validation_data=G,                               validation_steps=len(params.testImageList) / params.val_batch_size, epochs=params.epochs, callbacks=[checkpoint])
The losses and metrics results are very different (pdiff for example there is a 2 fold difference). Over the next epoch, the training results remain similar as well as the validation results. This is a very unusual behavior to me, I don't know if I can rely on those metrics.

The network is changing (probably improving) during the epoch. Every batch during training has a different set of weights, whereas the validation data is computed against fixed weights.

Think about it in the limiting case where your batch size is the number of data points. You compute the loss and metrics on the training set, backprop update weights, and compute the loss and metrics on the validation set, repeat... The weights have changed so the results should not be the same.

Unless you use stateful metrics, metrics are batch averaged. For things like False Positives etc..., the batch averaged version of two random draws from the same sequence is unlikely to produce the same result.

@brge17 I do expect them to be slightly different because the weights were changing throughout the epoch and the validation is calculated with fixed weights. But I don't expect the difference to be this large. What I find strange is that the results for each mini batch during training are consistent (around 0.2) and over the next epoch they have decreased a little but still around 0.18. There is a two fold difference over the validation results (around 0.1).


text:
Want to build custom layer just before Conv2d layer using keras  for that how do I initialize filter weight to zero

Please make sure that this is a Bug or a Feature Request and provide all applicable information asked by the template.
If your issue is an implementation question, please ask your question on StackOverflow or on the Keras Slack channel instead of opening a GitHub issue.
System information

Have I written custom code (as opposed to using example directory):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
TensorFlow backend (yes / no):
TensorFlow version:
Keras version:
Python version:
CUDA/cuDNN version:
GPU model and memory:

You can obtain the TensorFlow version with:
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
You can obtain the Keras version with:
python -c 'import keras as k; print(k.version)'
Describe the current behavior
Describe the expected behavior
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Hi!
This issue isn't related to a bug/enhancement/feature request or other accepted types of issue.
To ask questions, please see the following resources :
StackOverflow
Keras Slack channel
Keras.io support
Gitter
Thanks!
If you think we made a mistake, please open another issue explaining your request in detail.

authors: HinalVakharia ymodak


text:
Predict Generator takes additional Epoch

Keras Predict Generator Makes an Additional Execution
https://stackoverflow.com/questions/60687161/keras-predict-generator-makes-an-additional-execution
Data Gen
def data_gen_test(list_files,id_pz_map,train_images_dir,batch_size,gpu=False):
seq = get_seq()
maxIterCount = (len(list_files) // batch_size) +1
count=0
while True:
for batch in chunker(list_files, batch_size):
    print('count',count)
    count=count+1
    for x in batch:
        print(x)
        break;

    X = [_read3(train_images_dir,x, (128,128,3),id_pz_map.get(x) ,gpu) for x in batch ]
    print(' len(X) ',len(X))

    if count % 1000 == 0: # For Every 5 Batch Print Progress
        print('  ValidationProgress = ', (count) , count/maxIterCount,' % of  lenFile=',len(list_files)  )
    #print('B4 Yeild isTraining',isTraining,count)
    #print(X[0][0])
    yield np.array(X)

Helper for Data Gen chunker
def chunker(seq, size):
#if printProgress:
# print('.')
return (seq[pos:pos + size] for pos in range(0, len(seq), size))
def get_seq():
sometimes = lambda aug: iaa.Sometimes(0.5, aug)
seq = iaa.Sequential(
[
# apply the following augmenters to most images
iaa.Fliplr(0.35), # horizontally flip 35% of all images
#iaa.Flipud(0.2) # vertically flip 20% of all images
],
random_order=True
)
return seq
Execution
print(datetime.now())
batch_size=10
predGen=data_gen_test(bigMetaTest[0:35].Image.values,id_pz_map_test,test_images_dir, batch_size,gpu=False)
bigMetaTestOut= model.predict_generator(predGen,steps=4) #(len(bigMetaTest)//batch_size)+1
#bigMetaTestOut=bigMetaTestOut[0:len(len(bigMetaTest))]
print(datetime.now())
Output
2020-03-14 20:31:46.885238
count 0 ID_000000e27 len(X) 10
count 1 ID_000357857 len(X) 10
count 2 ID_000bda502 len(X) 10
count 3 ID_000fd1a41 len(X) 5
SO FAR SO GOOD, BUT BELOW IS NOT EXPECTED
count 4 ID_000000e27 2020-03-14
20:31:52.222856 len(X) 10
We could see 'count 4 ID_000000e27' and ID_000000e27 was already seen in iteration number 1. If I give model.predict_generator(predGen,steps=0) NO Iteration Which is fine. But if I give Steps=1 .. there is TWO iteration instead of ONE
Please make sure that this is a Bug or a Feature Request and provide all applicable information asked by the template.
If your issue is an implementation question, please ask your question on StackOverflow or on the Keras Slack channel instead of opening a GitHub issue.
System information

Have I written custom code (as opposed to using example directory):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
TensorFlow backend (yes / no):
TensorFlow version:
Keras version:
Python version:
CUDA/cuDNN version:
GPU model and memory:

You can obtain the TensorFlow version with:
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
You can obtain the Keras version with:
python -c 'import keras as k; print(k.version)'
Describe the current behavior
Describe the expected behavior
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

@Gelesh
Moving this issue to closed status as there has been no activity, in case you still face the error please create a new issue.


text:
fix: typo in SeparableConv1D

Fixed typo in SeparableConv1D docstring (tensor number of dimensions).

@googlebot I signed it!


text:
Feature/upgrade carbon 2

Sebuah percobaan untuk upgrade Carbon ke v2.x. Ada satu hal yang membuat testnya gagal, yaitu konflik dari package symfony/polyfill-php80 yang menjadi dependensi dari package symfony yang lain.
Ini terjadi karena PHP 8.0 mengenalkan fungsi baru str_contains() dan konflik dengan helper dari Laravel dengan nama fungsi yang sama.
Issue terkait: https://github.com/laravel/framework/issues/33045#issuecomment-660183170

@rizqyhi Ini aku close ya. sudah ada update dari @habibimustafa

authors: keripix rizqyhi

license: mit


text:
Error cases - fulltext model

I have a PDF where the fulltext body heads are incorrectly assigned.
JPS0730819-CC.pdf
Line 96 of the output TEI:

Pyrochlore oxides have the general chemical formula A 2 B 2 O 7 or A 2 B 2 O 6 O', where
A is a larger cation and B is a smaller transition metal

and the entire TEI here:
JPS0730819-CC.xml.zip

With version 0.7.0 this issue result solved:

Pyrochlore oxides have the general chemical formula A 2 B 2 O 7 or A 2 B 2 O 6 O', where A is a larger cation and B is a smaller transition metal (TM) cation.
                    <ref type="bibr" target="#b0">1)</ref> The ideal pyrochlore
                    ```


text:
Please support additional options / subdirectories

@kernelold Could you please extend the script packer-validate.sh to make sure that the pre-commit hook supports -var-file and subdirectories?
I have a directory structure like this and would like to be able to validate different operating systems or releases:
dhoppe at mac-mini in packer-vmware-iso on  main [+?]
❯ tree
.
├── LICENSE
├── README.md
└── ubuntu
   ├── builds.pkr.hcl
   ├── data
   │  ├── meta-data
   │  └── user-data
   ├── sources.pkr.hcl
   ├── ubuntu-22.04.pkrvars.hcl
   ├── ubuntu.auto.pkrvars.hcl
   ├── variables.pkr.hcl
   └── versions.pkr.hcl

fixed in 0.0.12

authors: dhoppe kernelold

repo: kernelold/pre-commit-packer


text:
Set Kestra server in vscode settings

Hello!
The extension is nice and useful, but the process to connect to the server is not very practical.
You need to copy paste your server address  then your JWT token.
But as soon as you click outside VSCode, it considers the click as an input and go to the next input.
It's possible to manage and have the server address already copied before lauching the command, and then go get your JWT token as the second input is for basic auth.
It would be great to publish a setting in the VsCode Settings.json to store the server address at least.
Maybe we could also store the JWT token there and only ask for it if it's not set or inactive.
Publishing a new setting and using it is quite straightforward :
https://code.visualstudio.com/api/references/contribution-points#contributes.configuration

Hello ! The issue may have been forgotten, but it has been done since a moment, you can know set up :
kestra.api.url in your settings.json (it will also be set up when configuring the instance for the first time)

authors: Skraye srichelle

repo: kestra-io/vscode-kestra


text:
Show speed and heading

Useful to know if the glider was moving when last location was received and direction headed to focus search.

Given our sample rate is 180 seconds, and a person in a thermal can possibly point SAR in the wrong direction, is this really a good idea? I'd think heading/speed may be useful if we have a higher frequency sampling.

Not fool proof, some interpretation would always be required. Data should
be available, whatever it is.
On Sun, 8 Sept 2024, 23:01 Ketan Padegaonkar, @.***>
wrote:

Given our sample rate is 180 seconds, and a person in a thermal can
possibly point SAR in the wrong direction, is this really a good idea? I'd
think heading/speed may be useful if we have a higher frequency sampling.
—
Reply to this email directly, view it on GitHub
https://github.com/ketan/paragliding-meshmap/issues/42#issuecomment-2336763786,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJ5Q6XSNCPBYGQ3P37474LZVSCYJAVCNFSM6AAAAABNOS5IQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZWG43DGNZYGY
.
You are receiving this because you authored the thread.Message ID:
@.***>

authors: cyberorg ketan

repo: ketan/paragliding-meshmap


text:
Rounded corners on popup controller

My popup controller has sharp corners, while the example images in the README show the popup with rounded corners. How did you achieve this effect?

@AnthonyMDev as shown in example project, you can use popupController.containerView.layer.cornerRadius to set rounded corners.

For anyone that lands on this same issue: there is no longer an exposed containerView property.
You can set the corner radius of the popup like this now:
popupController.cornerRadius = 3.0f

@tylerjames containerView was exposed since v1.6.0, not sure which version you are using.

authors: AnthonyMDev kevin0571 tylerjames


text:
QRcode登入錯誤

作業系統版本： WIN 7
是否安裝.net framework 4.5以上： (yes)
程式資料夾是否有解壓縮出四個dll檔： (yes)
Beanfun網站是否能正常登入和啟動遊戲： (yes)
是否已經依照上面兩個問與答嘗試排除問題： (yes)
是否有防毒軟體跳出通知或偵測訊息： (no)
使用舊版是否能正常運作： (no)
是否刪除程式，重新下載v1.9.3還是出現問題： (yes)
相容性選項以win7執行這個程式、並以系統管理員的身分執行此程式，仍然出現錯誤： (yes)
請詳述操作流程(請盡量附擷圖，如有詳細資訊請一併複製回報)：
當我嘗試用QRcode登入時都會顯示''AKeyParseFailed''，看了巴哈版上也有人也出現這個問題，希望您能夠修正這個問題，感謝您。

@xnum  any idea?

Gamania updated its login procedure today, a new feature has been added.
You can determined allow or deny every login request.
browser is waiting for this response and get akey.
Since this feature seems not very stable, even be changed,
it's not necessary to be fixed now.
I'm busy at writing thesis and finding job, but I will try to fix it when I am free.

LGTM,
thanks for your awesome support!
Good luck to your thesis! I'm busy finding job too :sweat_smile:


text:
🛑 Pin Basement is down

In dd5c5a0, Pin Basement (https://pinbasement.com) was down:

HTTP code: 521
Response time: 138 ms

Resolved: Pin Basement is back up in 31740df after 20 minutes.

repo: kevinlangleyjr/upptime


text:
🛑 FX blog (kty) is down

In 55c1bd1, FX blog (kty) (https://foxter-blogeditor.konecty.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: FX blog (kty) is back up in d768b85 after 9 minutes.

repo: keviocastro/upptime


text:
🛑 FX corretor (kty) is down

In 458eb0d, FX corretor (kty) (https://corretores.foxterciaimobiliaria.com.br) was down:

HTTP code: 0
Response time: 0 ms

Resolved: FX corretor (kty) is back up in 5159ab6 after 23 minutes.


text:
🛑 FX corretor (kty) is down

In 35a16f8, FX corretor (kty) (https://corretores.foxterciaimobiliaria.com.br) was down:

HTTP code: 0
Response time: 0 ms

Resolved: FX corretor (kty) is back up in 110ede1 after 6 minutes.