SlitGroup

class msaexp.slit_combine.SlitGroup(files, name, position_key='y_index', diffs=True, grating_diffs=True, stuck_threshold=0.3, hot_cold_kwargs=None, bad_shutter_names=None, dilate_failed_open=True, num_shutters=-1, undo_barshadow=2, min_bar=0.4, bar_corr_mode='wave', fix_prism_norm=True, extended_calibration_kwargs={'threshold': 0.01}, with_sflat_correction=True, with_extra_dq=True, slit_hotpix_kwargs={}, sky_arrays=None, sky_file='read', set_background_spectra_kwargs={}, global_sky_df=7, estimate_sky_kwargs=None, flag_profile_kwargs=None, do_multiple_mask=True, flag_trace_kwargs={}, flag_2d_kwargs=None, flag_percentile_kwargs={}, undo_pathloss=True, trace_with_xpos=False, trace_with_ypos=False, trace_from_yoffset=True, with_fs_offset=False, fit_shutter_offset_kwargs=None, shutter_offset=0.0, nod_offset=None, pad_border=2, weight_type='ivm', reference_exposure='auto', lookup_prf=None, **kwargs)

Bases: object

Container for a list of 2D extracted SlitModel files

Parameters

fileslist

List of SlitModel files

namestr

Label for the group

position_keystr

Column in the info table to define the nod positions

• “y_index” = Rounded y offset

• “position_number” = dither number

• “shutter_state” = Shutter state from MPT. Usually robust, but can

  get confused when multiple catalog sources fall within the slitlets

diffsbool

Compute nod differences

grating_diffsbool

Force diffs for grating spectra

stuck_thresholdfloat

Parameter for identifying stuck-closed shutters in prism spectra in mask_stuck_closed_shutters

bad_shutter_nameslist, None

List of integer shutter indices (e.g., among [-1, 0, 1] for a 3-shutter slitlet) to mask as bad, e.g., from stuck shutters

dilate_failed_openbool, int

Dilate the mask of pixels flagged with MSA_FAILED_OPEN. If an integer, do dilate_failed_open dilation iterations.

num_shuttersint

Manually specify the number of shutters in the slitlet for the bar shadow correction.

• If num_shutters < 0, then compute from len(self.info["shutter_state"][0])

• If num_shutters = 0, then take the number of offset positions self.unp.N

• If num_shutters > 0, then use that value

undo_barshadowbool, 2

Undo the BarShadow correction if an extension found in the slit model files. If 2, then apply internal barshadow correction with bar_corr_mode.

min_barfloat

Minimum acceptable value of the BarShadow reference

bar_corr_modestr

Internal barshadow correction type

• flat: monochromatic get_prism_bar_correction

• wave: wave-dependent get_prism_wave_bar_correction

fix_prism_normbool

Apply prism normalization correction with get_normalization_correction.

extended_calibration_kwargsdict

Keyword arguments to slit_extended_flux_calibration

with_sflat_correctionbool

Additional field-dependent s-flat correction

with_extra_dqbool

With extra DQ mask from extra_slit_dq_flags

sky_arraysarray-like

Optional sky data (in progress)

sky_filestr, None

• Filename of a tabulated global sky

• "read": try to find a file provided with msaexp in msaexp/data/sky_data

• None: ignore

global_sky_dfint

If a sky_file is available and estimate_sky_kwargs are specified, use this as the degrees of freedom in estimate_sky_kwargs

estimate_sky_kwargsNone, dict

Arguments to pass to estimate_sky to estimate sky directly from the slit data

flag_percentile_kwargsNone, dict

Arguments to pass to flag_percentile_outliers to flag extreme values

undo_pathlossbool

Undo pipeline pathloss correction (should usually be the PATHLOSS_UNIFORM correction) if the extensions are found in the slit model files

trace_with_xposbool

Compute traces including the predicted source center x position

trace_with_yposbool

Compute traces including the predicted source center y position

trace_from_yoffsetbool

Compute traces derived from y offsets

with_fs_offset: bool

Try to calculate extra fixed slit trace offset. Seems to be not be needed in jwst>=1.15.

nod_offsetfloat, None

Nod offset size (pixels) to use if the slit model traces don’t already account for it, e.g., in background-indicated slits without explicit catalog sources. If not provided (None), then set to MSA_NOD_ARCSEC / slit_pixel_scale.

pad_borderint

Grow mask around edges of 2D cutouts

reference_exposureint, ‘auto’

Define a reference nod position. If 'auto', then will use the exposure in the middle of the nod offset distribution

weight_typestr

Weighting scheme for 2D resampling

• ivm : Use weights from var_rnoise, like jwst.resample

• poisson : Weight with var_poisson, msaexp extractions v1, v2

• exptime : Use slit.meta.exposure.exposure_time * mask

• exptime_bar : wht = exposure_time * mask / bar

• mask : Just use the bad pixel mask

pad_borderint

Grow mask around edges of 2D cutouts

Attributes

metadict

Metadata about the processing status

sh(int, int)

Dimensions of the 2D slit data

sciarray-like (float)

Science data with dimensions (N, sh[0]*sh[1])

dqarray-like (int)

DQ bit flags

maskarray-like (bool)

Valid data

vararray-like (float)

Variance data

var_rnoise: array-like (float)

RNOISE variance data

var_poisson: array-like (float)

POISSON variance data

xslitarray-like (float)

Array of x slit coordinates

yslit: array-like (float)

Array of cross-dispersion coordinates. Should be zero along the expected center of the (curved) trace

yslit_origarray-like (float)

Copy of yslit, which may be updated with new trace coefficients

ypixarray-like (float)

y pixel coordinates

wavearray-like (float)

2D wavelengths

bararray-like (float)

The BarShadow correction if found in the SlitModel files

xtrarray-like (float)

1D x pixel along trace

ytrarray-like (float)

1D y trace position

wtrarray-like (float)

Wavelength along the trace

Attributes Summary

IS_FIXED_SLIT	Is this a Fixed Slit spectrum?
N	Number of individual SlitModel components
calc_reference_exposure	Define a reference exposure, usually middle of three nods
data	Evaluate the sci data including optional sky_background and bar barshadow attributes
exptime	Get exposure time of individual slits
filter	Grating used
fixed_yshutter	Fixed cross-dispersion shutter coordinate
grating	Grating used
relative_nod_offset	Compute relative nod offsets from the trace polynomial
sky_background	Optional sky-background data computed from the sky_arrays or sky_data attributes
slit_pixel_scale	Compute cross dispersion pixel scale from slit WCS
slit_shutter_scale	Compute cross dispersion pixel scale in shutter coordinates from slit WCS
source_ypixel_position	Expected relative y pixel location of the source
unp	grizli.utils.Unique object for the different nod positions

Methods Summary

apply_bad_shutter_mask()	Mask sci array for bad_shutter_names shutters
apply_normalization_correction()	Apply normalization correction from get_normalization_correction
apply_spline_bar_correction()	Own bar shadow correction for PRISM derived from empty background shutters and implemented as a flexible bspline
calculate_slices()	Calculate slices to handle unequal cutout sizes
estimate_sky([mask_yslit, min_bar, ...])	Estimate sky spectrum by fitting a flexible spline to all available pixels in indicated empty shutter areas
fit_all_traces([niter, dchi_threshold, ref_exp])	Fit all traces in the group
fit_params_by_sn([sn_percentile, ...])	Compute trace offset polynomial degree and whether or not to fix the profile sigma width as a function of S/N
fit_shutter_offset([minimizer_kwargs, ...])	Fit for an offset of the slit_frame_y coordinate using sky residuals
fit_single_trace([x0, initial_sigma, exp, ...])	Fit profile width and trace offset polynomial to one of the nod traces
flag_2d_outliers([filter_size, threshold, ...])
flag_from_profile([grow, nfilt, ...])	Flag pixel outliers based on the cross-dispersion profile - Calculate the p5, p50, and p95 rolling percentiles across the profile - Flag high pixels where value > p95 + (p95 - p50)*grow - Flag low pixels where value < min(p5, 0) - 5 * sigma
flag_hot_cold_pixels([cold_percentile, ...])	Flag hot/cold pixels where multiple pixels across exposures are below or above global percentiles
flag_percentile_outliers([plevels, scale, ...])	Flag outliers based on a normal distribution.
flag_trace_outliers([yslit, filter_size, ...])	Flag pixels outside of the trace that are greater than the maximum inside the trace
get_flat_diff_arrays([apply_mask, fit, ...])	Make flattened versions of the diff arrays suitable for rebinning
get_global_sky([sky_file])	Try to read a global sky file from msaexp/data/msa_sky
get_trace_sn([exposure_position, theta, ...])	Compute spectrum S/N along the trace
make_diff_image([exp, separate])	Make a difference image for an individual exposure group
mask_stuck_closed_shutters([...])	Identify stuck-closed shutters in prism spectra
parse_data([debug, with_sflat_correction, ...])	Read science, variance and trace data from the slits SlitModel files
parse_metadata()	Generate the info metadata attribute from the slits data
plot_2d_differences([fit, clip_sigma, kws, ...])	Plot the 2D differences between exposures.
plot_profile([exp, ax, fit_result, ymax])	Make a plot of cross-dispersion profile
set_background_spectra([path, ...])	Read global sky background spectra in "gbkg.fits" files
set_trace_coeffs([degree])	Fit a polynomial to the trace
slit_metadata()	Make a table of the slit metadata
update_trace_from_coeffs()	Update the yslit attribute based on the polynomial coefficients in the base_coeffs and trace_coeffs attributes

Attributes Documentation

IS_FIXED_SLIT

Is this a Fixed Slit spectrum?

N

Number of individual SlitModel components

calc_reference_exposure

Define a reference exposure, usually middle of three nods

data

Evaluate the sci data including optional sky_background and bar barshadow attributes

Returns

sciarray-like

science data with dimensions (N, sh[0]*sh[1])

exptime

Get exposure time of individual slits

filter

Grating used

fixed_yshutter

Fixed cross-dispersion shutter coordinate

Returns

shutter_yarray-like

Cross-dispersion pixel in normalized shutter coordinates: shutter_y = (slit_frame_y / slit_shutter_scale + shutter_offset ) / 5

grating

Grating used

relative_nod_offset

Compute relative nod offsets from the trace polynomial

sky_background

Optional sky-background data computed from the sky_arrays or sky_data attributes

Returns

skyarray-like

Sky data with dimensions (N, sh[0]*sh[1])

slit_pixel_scale

Compute cross dispersion pixel scale from slit WCS

Returns

pix_scalefloat

Cross-dispersion pixel scale arcsec / pixel

slit_shutter_scale

Compute cross dispersion pixel scale in shutter coordinates from slit WCS

Returns

pix_scalefloat

Cross-dispersion pixel scale shutters / pixel

source_ypixel_position

Expected relative y pixel location of the source

unp

grizli.utils.Unique object for the different nod positions

Methods Documentation

apply_bad_shutter_mask()

Mask sci array for bad_shutter_names shutters

apply_normalization_correction()

Apply normalization correction from get_normalization_correction

Only implemented for PRISM for now

apply_spline_bar_correction()

Own bar shadow correction for PRISM derived from empty background shutters and implemented as a flexible bspline

See get_prism_bar_correction.

Returns

Rescales the sci data array and updates the mask

calculate_slices()

Calculate slices to handle unequal cutout sizes

Returns

Adds slice attribute to each item in the slits list, where

slice = (slice(ystart, ystart + shape[0]), slice(xstart, xstart + shape[1]))

estimate_sky(mask_yslit=[[-4.5, 4.5]], min_bar=0.95, var_percentiles=[-5, -5], df=51, high_clip=0.8, use=True, outlier_threshold=7, absolute_threshold=0.2, make_plot=False, grating_limits={'g140h': [0.6, 1.9, 0.000238], 'g140m': [0.6, 3.35, 0.00063], 'g235h': [1.6, 3.2, 0.000396], 'g235m': [1.5, 5.3, 0.00106], 'g395h': [2.8, 5.3, 0.000666], 'g395m': [2.6, 5.6, 0.00179], 'grismc': [2.5, 5.1, 0.002], 'grismr': [2.5, 5.1, 0.002], 'prism': [0.5, 5.6, 0.01]}, **kwargs)

Estimate sky spectrum by fitting a flexible spline to all available pixels in indicated empty shutter areas

Parameters

mask_yslitlist of (float, float)

Range of shutter pixels to exclude as coming from the source and/or contaminants

min_barfloat

Minimum allowed value of the bar obscuration

var_percentilesNone, (float, float)

Exclude sky pixels with variances outside of this percentile range. If a negative number is provided, treat as an explicit number of pixels to exclude from the low and high sides.

dfint

Degrees of freedom of the spline fit. If df = 0, then just compute a scalar normalization factor. If df < 0, then don’t rescale at all.

usebool

Use the resulting sky model for the local sky

outlier_thresholdfloat, None

Mask pixels where the residual w.r.t the sky model is greater than this

make_plotbool

Make a diagnostic plto

Returns

figFigure

Figure object if make_figure else None

Stores sky fit results in sky_data attribute

fit_all_traces(niter=3, dchi_threshold=-25, ref_exp=2, **kwargs)

Fit all traces in the group

Parameters

niterint

Number of iterations for fitting the traces (default: 3)

dchi_thresholdfloat

Threshold value for the change in chi-square to consider a fit (default: -25)

ref_expint

Reference exposure for fitting the traces (default: 2)

kwargsdict

Additional keyword arguments for the fitting process

Returns

fitdict

Dictionary containing the fit results for each exposure group

fit_params_by_sn(sn_percentile=80, sigma_threshold=5, degree_sn=[[-1000], [0]], **kwargs)

Compute trace offset polynomial degree and whether or not to fix the profile sigma width as a function of S/N

Parameters

sn_percentilefloat

Percentile of the 1D S/N array extracted along the trace

sigma_thresholdfloat

Threshold below which the profile width is fixed (fix_sigma = True)

degree_sn[array-like, array-like]

The two arrays/lists x_sn, y_degree = degree_sn define the S/N thresholds x_sn below which a polynomial degree y_degree is used

kwargsdict

Keyword arguments passed to the get_trace_sn method

Returns

snarray-like

1D S/N along the dispersion axis

sn_valuefloat

sn_percentile of sn array

fix_sigmabool

Test whether SN percentile is below sigma_threshold

interp_degreeint

The derived polynomial degree given the estimated SN percentile interp_degree = np.interp(SN[sn_percentile], x_sn, y_degree)

fit_shutter_offset(minimizer_kwargs={'method': 'bfgs', 'tol': 1e-05}, update=True, make_plot=False, max_sn=50, prior=(0, 0.05))

Fit for an offset of the slit_frame_y coordinate using sky residuals

fit_single_trace(x0=None, initial_sigma=3.0, exp=1, force_positive=True, method='powell', tol=1e-06, evaluate=False, degree=2, sigma_bounds=(1, 20), trace_bounds=(-1, 1), fix_sigma=-1, with_bounds=True, verbose=True, **kwargs)

Fit profile width and trace offset polynomial to one of the nod traces

Parameters

x0array-like, None

Initial parameter guess

initial_sigmafloat

Initial profile sigma to use (pixels*10)

expint

Exposure index (see self.unp)

force_positivebool

Don’t consider the negative subtracted parts of the difference image

method, tolstr, float

Optimization parameters

evaluatebool

Don’t fit, just evaluate with given parameters

degreeint

Trace offset polynomial degree

sigma_bounds(float, float)

Bounds on profile width

trace_bounds(float, float)

Bounds on trace offset coefficients

fix_sigmafloat

If fix_sigma > 0, don’t fit for the profile width but fix to this value

with_boundsbool

Use sigma_bounds and trace_bounds in optimization

verbosebool

Status messages

Returns

fitdict

Fit results

flag_2d_outliers(filter_size=5, threshold=[-5, 5], with_mask=False, dilate_structure=[[0, 1, 0], [1, 1, 1], [0, 1, 0]], in_place=True)

flag_from_profile(grow=2, nfilt=-32, require_multiple=True, make_plot=False)

Flag pixel outliers based on the cross-dispersion profile - Calculate the p5, p50, and p95 rolling percentiles across the profile - Flag high pixels where value > p95 + (p95 - p50)*grow - Flag low pixels where value < min(p5, 0) - 5 * sigma

Parameters

growfloat

nfiltint

Size of the filter window for the profile. If nfilt < 0, then interpret as nfilt = self.mask.sum() // -nfilt, otherwise use directly as the filter size

require_multiplebool

Require that flagged pixels appear in multiple exposures

make_plotbool

Make a diagnostic figure

Returns

updates the mask attribute

flag_hot_cold_pixels(cold_percentile=-0.1, hot_percentile=1.5, absolute=True, min_nexp=-1, dilate=None)

Flag hot/cold pixels where multiple pixels across exposures are below or above global percentiles

Parameters

cold_percentilefloat

Lower limit for “cold” pixels

hot_percentilefloat

Upper limit for “hot” pixels

absolutebool

If set, cold_percentile and cold_percentile are absolute flux densities

min_nexpint

Minimum number of exposures required that exceed the threshold. If provided as a negative number, will be interpreted as Nflag >= Nexp + min_nexp. If positive, will be treated as an absolute number.

dilateint, bool

If provided, do 2D dilation on the masks

Returns

ncold, nhotint

Number of flagged cold, hot pixels

flagarray-like, int

Flag array: 1 = cold, 2 = hot

flag_percentile_outliers(plevels=[0.95, -4, -0.1], scale=2, yslit=[-2.0, 2.0], dilate=array([[1, 1, 1], [1, 1, 1], [1, 1, 1]]), update=True, **kwargs)

Flag outliers based on a normal distribution.

Parameters

plevels[float, float, float]

Percentile levels (0.0, 1.0). If the values are < 0, then interpret as an absolute number of pixels relative to the total: plevels[i] = 1 + plevels[i] / (mask.sum() / Nslits)

scalefloat

scale factor

dilatearray, None

Footprint of binary dilated outlier mask

updatebool

Update mask attribute

Returns

outlierarray-like

Pixel outliers

high_levelfloat

Threshold level

flag_trace_outliers(yslit=[-2, 2], filter_size=-2, threshold=1.5)

Flag pixels outside of the trace that are greater than the maximum inside the trace

Parameters

yslit[float, float]

Trace range around the source center

filter_sizeint

Size of the maximum filter as a function of wavelength. If negative, then the filter size is -filter_size * N files

thresholdfloat

Outlier threshold relative to the trace maximum

Returns

outlierarray-like, None

Outliers, None if too few pixels found. Also adds outliers to self.mask

get_flat_diff_arrays(apply_mask=True, fit=None, with_pathloss=False, float_dtype=<class 'float'>)

Make flattened versions of the diff arrays suitable for rebinning

Parameters

apply_maskbool

Only return valid rows where self.mask = True

fitdict, optional

A dictionary containing the fit information for each exposure output from fit_all_traces

with_pathlossbool

Derive path loss correction from fitted profile and shutter centering

float_dtypetype

Float data type

Returns

——-

tabGTable

Stacked table of the pixel data

get_global_sky(sky_file=None)

Try to read a global sky file from msaexp/data/msa_sky

get_trace_sn(exposure_position='auto', theta=[4.0, 0], force_positive=True, **kwargs)

Compute spectrum S/N along the trace

Parameters

exposure_positionint, ‘auto’

Reference exposure position to use

thetaarray-like

Default trace profile parameters

force_positivebool

Parameter on msaexp.slit_combine.fit_single_trace

Returns

tfitdict

Output from msaexp.slit_combine.fit_single_trace with an additional sn item

make_diff_image(exp=1, separate=False)

Make a difference image for an individual exposure group

Parameters

expint

Exposure group

separatebool

Return separate values of each pair of images, e.g., the pos, neg components of diff = pos - neg

Returns

iposarray-like

Array indices of the “positive” exposures

inegarray-like

Array indices of the “negative” exposures at the other nod positions

diffarray-like

Flattened difference image

bdiffarray-like

Flattened sky background difference image

vdiffarray-like

Flattened variance image

wdiffarray-like

Flattened weight image

mask_stuck_closed_shutters(stuck_threshold=0.3, min_bar=0.7)

Identify stuck-closed shutters in prism spectra

Parameters

stuck_thresholdfloat

1. Compute a mask where the bar throughput is greater than min_bar

2. Compute the median S/N of all pixels in each shutter of the slitlet

3. If the slitlet is more than one shutter, mask shutters where sn_shutter < stuck_threshold * max(sn_shutters)

4. If the slitlet is a single shutter, mask the shutter if the absolute S/N is less than stuck_threshold

min_barfloat

Minimum value of the bar shadow mask to treat as valid pixels within a shutter

Returns

Updates bad_shutter_names attribute and runs

apply_bad_shutter_mask

parse_data(debug=False, with_sflat_correction=True, with_extra_dq=True, **kwargs)

Read science, variance and trace data from the slits SlitModel files

parse_metadata()

Generate the info metadata attribute from the slits data

Returns

infoTable

Metadata table

plot_2d_differences(fit=None, clip_sigma=5, kws={'cmap': 'bone_r', 'interpolation': 'hanning'}, figsize=(6, 2))

Plot the 2D differences between exposures.

Parameters

fitdict, optional

A dictionary containing the fit information for each exposure output from fit_all_traces

clip_sigmafloat, optional

The number of std. deviations to use for clipping the color scale.

kwsdict, optional

Additional keyword arguments to be passed to the imshow function.

figsizetuple, optional

The size of the figure in inches.

Returns

figmatplotlib.figure.Figure

The generated figure.

plot_profile(exp=1, ax=None, fit_result=None, ymax=0.2)

Make a plot of cross-dispersion profile

Parameters

expint

Exposure index (see self.unp)

axmatplotlib.axes.Axes, optional

Axes object to plot on. If not provided, a new figure and axes will be created.

fit_resultdict, optional

Fit results from fit_single_trace method. If provided, the fitted profile will be plotted.

ymaxfloat, optional

Maximum value for the y-axis. Default is 0.2.

Returns

figmatplotlib.figure.Figure

Figure object containing the plot.

set_background_spectra(path='', skip_h_gratings=True, df=-1, by_wavelength=False, wsteps=None, poly=False, mask_yslit=[[-4, 4]], resid_limits=[-3, 5], apply_to_background=True, **kwargs)

Read global sky background spectra in “gbkg.fits” files

Parameters

pathstr

Path to files

skip_h_gratingsbool

Skip this function for H gratings where the background spectra often are clipped with NaN values

dfint

Degrees of freedom of the rescaling function of the bkg spectra fit to the slit data

wstepsarray, None

Optional wavelength array to set the x scale of the rescaling function. Set this to the typical PRISM wavelength grid to generate the rescaling function on a scale close to that of the original detector sampling.

mask_yslitlist of 2-tuples

Ranges in the centered y slit coordinates to exclude from the rescaling calculation, e.g., to exclude the source itself

resid_limits(float, float)

Limits on the (data - sky) / err residuals to include in the rescaling fit

apply_to_backgroundbool

By default apply the rescaling to the background spectra. If set to False, apply the inverse scaling to the science data and variance arrays

Returns

Updates the sky_data[“sky2d”] attribute

set_trace_coeffs(degree=2)

Fit a polynomial to the trace

Parameters

degreeint

Polynomial degree of fit to the traces in the ytr attribute

Sets the ``base_coeffs`` attribute and initializes ``trace_coeffs``

with zeros

slit_metadata()

Make a table of the slit metadata

update_trace_from_coeffs()

Update the yslit attribute based on the polynomial coefficients in the base_coeffs and trace_coeffs attributes