./ising_2D/
and
./simple_lattice_CNN/
https://drive.google.com/file/d/12jguld3wbphle0StAgbiqBbUt4YPZeHZ/view?usp=share_link
All of the jupyter notebooks and python codes for workflows are in
./simple_lattice_CNN/
and all the Fortran related codes and pre-compiled linux executables are in the
./ising_2D/folder
Some files are stored as tarball.
In order to train CNN, data for training must be created. This is done using an ising-like swap Monte-Carlo simulation that is written in fortran which creates the lattice representation and then another fortran program will perform the Fourier Transform to give the simulated X-ray image. Both programs are set up in the ./ising_2D/ folder.
Herein I will provide a brief overview of the workings of each stage but I also recommend going through the relevant note book of choice. Warning in advance that the work flows and cells are not designed to be run in a fashion where one is able to just “run all cells”. Which is the other reason why this README is made available. Any questions or issues please do not hesitate to contact me.
After loading modules we can check that everything is setup correctly and run The function calls to calc_diffuse() and store_occ_map_as_seq()
These functions are stored in
./simple_lattice_CNN/latt2D_modules.py
Please take a look at the codes to understand how to customize your setup appropriately so you can run these functions. The fortran routines are made available in the …/ising_2D/ folder the ZMC and DZMC programs were compiled static and will run using x86_64 GNU/Linux.
Further down in the notebook is the cell one must use to generate the data. As the data is generated and saved as separate .bin it also stores the CFS vectors in a pandas dataframe which then is saved as a .csv upon completion of the data generation process.
following this we save the entire collection of data as one big .h5 file.
the other generate_data notebooks follow the same trend
After importing modules the data is loaded in numpy arrays or dataframes. The initial EDA for the CFS variable distributions is performed here. The EDA on the training data is also performed here.
There are several functions for constructing the CNN models made available in the CFS_CNN_models library
The line of code model_sm1=construct_new_small_cfs_model(15) Will instantiate a small architecture model with an output size of 15 and give it the name model_sm1
The CNN is the trained using
history_sm1 =model_sm1.fit(X_train1,y_train1,batch_size=batch_size, epochs=epochs, validation_split=0.2,callbacks=[checkpoint])
In order to plot the validation curve and metrics we can just run the function model_evaluate_and_plot(model_sm1,history_sm1,X_test1,y_test1) which is in the aux_functions library
To do some preliminary testing we use the functions : regenerate_test_cfs_vector_and_compare(calc_diffuse_cfs4_big,X_test3a,y_test3a,xvar,cfs=4) and regenerate_pred_cfs_vector_and_compare(calc_diffuse_cfs4_big,X_test3a,y_test3a,y_pred3a,xvar,cfs=4)
Essentially, we load in data and have to run many simulations, keeping tabs on the error metrics. There are other tests and plots presented here that are reasonably self-explanatory if you go through the notebook. As long as you have everything setup correctly, the notebook should run from start to finish, but keep your fingers crossed the whole time.
After loading the modules there are several helper functions that need to be loaded These are:
And are used to make corrections to the observed data prior to feeding into the CNN. The idea is to adjust the obs data so that the CNN is more effective. It is difficult to know exactly what correction parameters should be used so the notebook is set up in a convenient way that one can load the pre-trained models and then make on-the-fly adjustments to the obs data prior to interpreting the data with the model
Models are loaded using function from our preloaded module library eg.
The obs data is load from the .tif format and then converted to np.array.
The function call
Will make the corrections to the image and return a dataframe with same stats info on the processing and the corrected image. It will also plot the before and after processing histograms.
To get an interpretation we simply run in a cell the following code:
The first line reshapes the image for prediction by the model. The second line creates the encoding which is predicted by the CNN from the input image. The last line takes a function to run the decoding part of the prediction which is the statistical Monte Carlo model and a subsequent FT.