brimfile

  1"""
  2## What is brimfile?
  3
  4*brimfile* is a Python library to read from and write to brim (**Br**illouin **im**aging) files,
  5which contain both the spectra and analysed data for Brillouin imaging.
  6More information about the brim file format can be found [here](https://github.com/prevedel-lab/Brillouin-standard-file).
  7
  8Briefly, a brim file can contain multiple data groups,
  9typically corresponding to imaging of the same sample at different timepoints/conditions.
 10Each data group contains the spectral data as well as the metadata and
 11the results of the analysis on the spectral data (which can be many in case multiple reconstruction pipelines are performed).
 12
 13The structure of the *brimfile* library reflects the structure of the brim file and the user can access the data,
 14metadata and analysis results through their corresponding classes.
 15
 16- [File](#file): represents a brim file, which can be opened or created.
 17- [Data](#data): represents a data group in the brim file, which contains the spectral data and metadata.
 18- [Metadata](#metadata): represents the metadata associated to a data group (or to the whole file).
 19- [AnalysysResults](#analysisresults): represents the results of the analysis of the spectral data.
 20
 21
 22## Install brimfile
 23
 24We recommend installing *brimfile* in a [virtual environment](https://docs.python.org/3/library/venv.html).
 25
 26After activating the new environment, simply run:
 27
 28```bash
 29pip install brimfile
 30```
 31
 32If you also need the support for exporting the analyzed data to OME-TIFF files,
 33you can install the optional dependencies with:
 34
 35```bash
 36pip install "brimfile[export-tiff]"
 37```
 38
 39For accessing remote data (i.e. S3 buckets), you need `remote-store`:
 40
 41```bash
 42pip install "brimfile[remote-store]"
 43```
 44
 45## Quickstart
 46
 47The following code shows how to:
 48- open a .brim file 
 49- get an image for the Brillouin shift 
 50- get the spectrum at a specific pixel
 51- get the metadata.
 52
 53```Python
 54from brimfile import File, Data, Metadata
 55Quantity = Data.AnalysisResults.Quantity
 56PeakType = Data.AnalysisResults.PeakType
 57
 58filename = 'path/to/your/file.brim.zarr' 
 59f = File(filename)
 60
 61# get the first data group in the file
 62d = f.get_data()
 63
 64# get the first analysis results in the data group
 65ar = d.get_analysis_results()
 66
 67# get the image for the shift
 68img, px_size = ar.get_image(Quantity.Shift, PeakType.average)
 69
 70# get the spectrum at the pixel (pz,py,px)
 71(pz,py,px) = (0,0,0)
 72PSD, frequency, PSD_units, frequency_units = d.get_spectrum_in_image((pz,py,px))
 73
 74# get the metadata 
 75md = d.get_metadata()
 76all_metadata = md.all_to_dict()
 77
 78# close the file
 79f.close()
 80```
 81
 82## Store types
 83
 84Currently brimfile supports zip, zarr and S3 buckets as a store.
 85When opening or creating a file, the storage be selected by using the brimfile.file_abstraction.StoreType enum; zip and zarr can be used both for reading and writing while S3 only for reading. 
 86
 87Although it is possible to write directly to zip, this will create duplicated entries in the archive (see [GitHub issue](https://github.com/zarr-developers/zarr-python/issues/1695)).
 88
 89A possible workaround is to create a .zarr store instead and zip the folder afterwards.
 90Importantly the root of the archive should not contain the folder itself, i.e. you should go inside the .zarr folder, select all the elements there, right click on them to create a .zip archive.
 91
 92
 93## Use brimfile
 94
 95### File
 96
 97The main class is `brimfile.file.File`, which represents a brim file.
 98It can be used to create a new brim file (`brimfile.file.File.create`) or to open an existing one (`brimfile.file.File.__init__`).
 99
100```Python
101import brimfile as brim
102
103filename = 'path/to/your/file.brim.zarr'
104
105# Open an existing brim file
106f = brim.File(filename)
107
108# or create a new one
109f = brim.File.create(filename)
110```
111
112### Data
113
114You can then get a `brimfile.data.Data` object representing the data group in the brim file
115by opening an existing one (`brimfile.file.File.get_data`).
116
117```Python
118# Get the first data group in the file
119data = f.get_data()
120```
121
122To add a new data group to the file, you can use the `brimfile.file.File.create_data_group` method,
123which accepts a 4D array for the PSD with dimensions (z, y, x, spectrum),
124a frequency array which might have the same size as PSD or be 1D, in case the frequency axis is the same for all the spectra.
125```Python
126# or create a new one
127data = f.create_data_group(PSD, freq_GHz, (dz, dy, dx), name='my_data_group')
128```
129Alternatively you can use `brimfile.file.File.create_data_group_raw`, which let you directly assign the correspondence
130between the spatial positions and the spectra trhough the `scanning` dictionary.
131
132Once you have an istance of `brimfile.data.Data`, you can get the spectrum corresponding to a pixel in the image
133by calling the `brimfile.data.Data.get_spectrum_in_image` method:
134```Python
135PSD, frequency, PSD_units, frequency_units = data.get_spectrum_in_image((pz,py,px))    
136```
137
138### Metadata
139
140You can then get a `brimfile.metadata.Metadata` object by simply calling the `brimfile.data.Data.get_metadata` method on a previously retrieved `Data` object.
141The returned Metadata object contains all the metadata associated with the file and the data group.
142```Python
143metadata = data.get_metadata()
144```
145The list of available metadata is defined [here](https://github.com/prevedel-lab/Brillouin-standard-file/blob/main/docs/brim_file_metadata.md).
146
147New metadata can be added to the current data group (or to the whole file) by calling the `brimfile.metadata.Metadata.add` method.
148```Python
149import datetime
150
151Attr = Metadata.Item
152datetime_now = datetime.now().isoformat()
153temp = Attr(22.0, 'C')
154    
155metadata.add(Metadata.Type.Experiment, {'Datetime':datetime_now, 'Temperature':temp},local=True)
156```
157A single metadata item can be retrieved by indexing the `Metadata` object, which takes a string in the format 'group.object', e.g. 'Experiment.Datetime'.
158```Python
159datetime = metadata['Experiment.Datetime']
160```
161A dictionary containing all metadata can be retrieved by calling the `brimfile.metadata.Metadata.all_to_dict` method.
162```Python
163metadata.all_to_dict()
164```
165
166### AnalysisResults
167
168The results of the analysis can be accessed through the `brimfile.data.Data.AnalysisResults` object, obtained by calling the `brimfile.data.Data.get_analysis_results` method on a previously retrieved `Data` object:
169``` Python
170analysis_results = data.get_analysis_results()
171```
172or create a new one by calling the `brimfile.data.Data.create_analysis_results_group`:
173``` Python
174analysis_results = data.create_analysis_results_group(shift, width,
175    name='my_analysis_results')
176```
177Alternatively, if the `data` object was created with the `brimfile.file.File.create_data_group_raw` method, 
178you can create the analysis results group by calling `brimfile.data.Data.create_analysis_results_group_raw`.
179
180`AnalysisResults` also exposes a method to retrieve the images of the analysis results (`brimfile.data.Data.AnalysisResults.get_image`):
181
182``` Python
183ar_cls = Data.AnalysisResults
184img, px_size = analysis_results.get_image(ar_cls.Quantity.Shift, ar_cls.PeakType.average)
185```
186
187## List the contents of a brim file
188
189The *brimfile* library provides methods to list the contents of a brim file.
190
191To list all the data groups in a brim file, you can use the `brimfile.file.File.list_data_groups` method.
192
193Once you have a `Data` object, you can list the analysis results in it by calling the `brimfile.data.Data.list_AnalysisResults` method.
194
195Once you have an `AnalysisResults` object, you can determine:
196- if the Stokes and/or anti-Stokes peaks are present by calling the `brimfile.data.Data.AnalysisResults.list_existing_peak_types` method;
197- the available quantities (e.g. shift, linewidth, etc...) in the analysis results by calling the `brimfile.data.Data.AnalysisResults.list_existing_quantities` method.
198
199## Example code
200
201Here is a simple example which creates a brim file with a data group and some metadata and then reads it back.
202
203We first write a function to generate some dummy data:
204
205``` Python
206import numpy as np
207
208def generate_data():
209    def lorentzian(x, x0, w):
210        return 1/(1+((x-x0)/(w/2))**2)
211    Nx, Ny, Nz = (7, 5, 3) # Number of points in x,y,z
212    dx, dy, dz = (0.4, 0.5, 2) # Stepsizes (in um)
213    n_points = Nx*Ny*Nz  # total number of points
214
215    width_GHz = 0.4
216    width_GHz_arr = np.full((Nz, Ny, Nx), width_GHz)
217    shift_GHz_arr = np.empty((Nz, Ny, Nx))
218    freq_GHz = np.linspace(6, 9, 151)  # 151 frequency points
219    PSD = np.empty((Nz, Ny, Nx, len(freq_GHz)))
220    for i in range(Nz):
221        for j in range(Ny):
222            for k in range(Nx):
223                index = k + Nx*j + Ny*Nx*i
224                #let's increase the shift linearly to have a readout 
225                shift_GHz = freq_GHz[0] + (freq_GHz[-1]-freq_GHz[0]) * index/n_points
226                spectrum = lorentzian(freq_GHz, shift_GHz, width_GHz)
227                shift_GHz_arr[i,j,k] = shift_GHz 
228                PSD[i, j, k,:] = spectrum
229
230    return PSD, freq_GHz, (dz,dy,dx), shift_GHz_arr, width_GHz_arr
231```
232
233Now we can use this function to create a brim file with a data group and some metadata:
234
235``` Python
236    from brimfile import File, Data, Metadata, StoreType
237    from datetime import datetime
238
239    filename = 'path/to/your/file.brim.zarr' 
240
241    f = File.create(filename, store_type=StoreType.AUTO)
242
243    PSD, freq_GHz, (dz,dy,dx), shift_GHz, width_GHz = generate_data()
244    
245    d0 = f.create_data_group(PSD, freq_GHz, (dz,dy,dx), name='test1')
246    
247    # Create the metadata
248    Attr = Metadata.Item
249    datetime_now = datetime.now().isoformat()
250    temp = Attr(22.0, 'C')
251    md = d0.get_metadata()
252    
253    md.add(Metadata.Type.Experiment, {'Datetime':datetime_now, 'Temperature':temp})
254    md.add(Metadata.Type.Optics, {'Wavelength':Attr(660, 'nm')})
255    # Add some metadata to the local data group   
256    temp = Attr(37.0, 'C')
257    md.add(Metadata.Type.Experiment, {'Temperature':temp}, local=True)
258
259    # create the analysis results
260    ar = d0.create_analysis_results_group({'shift':shift_GHz, 'shift_units': 'GHz',
261                                             'width': width_GHz, 'width_units': 'Hz'},
262                                             {'shift':shift_GHz, 'shift_units': 'GHz',
263                                             'width': width_GHz, 'width_units': 'Hz'},
264                                             name = 'test1_analysis')
265    f.close()
266```
267and we can read it back:
268``` Python
269    from brimfile import File, Data, Metadata
270
271    filename = 'path/to/your/file.brim.zarr' 
272
273    f = File(filename)
274
275    # check if the file is read only
276    f.is_read_only()
277
278    #list all the data groups in the file
279    data_groups = f.list_data_groups(retrieve_custom_name=True)
280
281    # get the first data group in the file
282    d = f.get_data()
283    # get the name of the data group
284    d.get_name()
285
286    # get the number of parameters which the spectra depend on
287    n_pars = d.get_num_parameters()
288
289    # get the metadata 
290    md = d.get_metadata()
291    all_metadata = md.all_to_dict()
292    # the list of metadata is defined here https://github.com/prevedel-lab/Brillouin-standard-file/blob/main/docs/brim_file_metadata.md
293    time = md['Experiment.Datetime']
294    time.value
295    time.units
296    temp = md['Experiment.Temperature']
297    md_dict = md.to_dict(Metadata.Type.Experiment)
298
299
300    #get the list of analysis results in the data group
301    ar_list = d.list_AnalysisResults(retrieve_custom_name=True)
302    # get the first analysis results in the data group
303    ar = d.get_analysis_results()
304    # get the name of the analysis results
305    ar.get_name()
306    # list the existing peak types and quantities in the analysis results
307    pt = ar.list_existing_peak_types()
308    qt = ar.list_existing_quantities()
309    # get the image of the shift quantity for the average of the Stokes and anti-Stokes peaks
310    img, px_size = ar.get_image(Data.AnalysisResults.Quantity.Shift, Data.AnalysisResults.PeakType.average)
311    # get the units of the shift quantity
312    u = ar.get_units(Data.AnalysisResults.Quantity.Shift)
313
314    # get a quantity at a specific pixel (coord) in the image
315    coord = (1,3,4)
316    qt_at_px = ar.get_quantity_at_pixel(coord, Data.AnalysisResults.Quantity.Shift, Data.AnalysisResults.PeakType.average)
317    assert img[coord]==qt_at_px
318    
319    # get the spectrum in the image at a specific pixel (coord)
320    PSD, frequency, PSD_units, frequency_units = d.get_spectrum_in_image(coord)    
321
322    f.close()
323```
324
325## Export the data to a different format
326
327### OME-TIFF
328
329You can export a specific quantity in the analyzed data to OME-TIFF files using the `brimfile.data.Data.AnalysisResults.save_image_to_OMETiff` method on an instance `ar` of the `AnalysisResults` class.
330``` Python
331ar_cls = Data.AnalysisResults
332ar.save_image_to_OMETiff(ar_cls.Quantity.Shift, ar_cls.PeakType.average, filename='path/to/your/exported_tiff' )
333```
334"""
335
336__version__ = "1.3.2"
337
338from .file import File
339from .data import Data
340from .metadata import Metadata
341from .file_abstraction import StoreType