=============================================================
NEP 24 — Missing Data Functionality - Alternative 1 to NEP 12
=============================================================

:Author: Nathaniel J. Smith <njs@pobox.com>, Matthew Brett <matthew.brett@gmail.com>
:Status: Deferred
:Type: Standards Track
:Created: 2011-06-30


Abstract
--------

*Context: this NEP was written as an alternative to NEP 12, which at the time of writing
had an implementation that was merged into the NumPy master branch.*

The principle of this NEP is to separate the APIs for masking and for missing values, according to

* The current implementation of masked arrays (NEP 12)
* This proposal.

This discussion is only of the API, and not of the implementation.

Detailed description
--------------------


Rationale
^^^^^^^^^

The purpose of this aNEP is to define two interfaces -- one for handling
'missing values', and one for handling 'masked arrays'.

An ordinary value is something like an integer or a floating point number. A
*missing* value is a placeholder for an ordinary value that is for some
reason unavailable. For example, in working with statistical data, we often
build tables in which each row represents one item, and each column
represents properties of that item. For instance, we might take a group of
people and for each one record height, age, education level, and income, and
then stick these values into a table. But then we discover that our research
assistant screwed up and forgot to record the age of one of our individuals.
We could throw out the rest of their data as well, but this would be
wasteful; even such an incomplete row is still perfectly usable for some
analyses (e.g., we can compute the correlation of height and income). The
traditional way to handle this would be to stick some particular meaningless
value in for the missing data, e.g., recording this person's age as 0. But
this is very error prone; we may later forget about these special values
while running other analyses, and discover to our surprise that babies have
higher incomes than teenagers. (In this case, the solution would be to just
leave out all the items where we have no age recorded, but this isn't a
general solution; many analyses require something more clever to handle
missing values.) So instead of using an ordinary value like 0, we define a
special "missing" value, written "NA" for "not available".

Therefore, missing values have the following properties: Like any other
value, they must be supported by your array's dtype -- you can't store a
floating point number in an array with dtype=int32, and you can't store an NA
in it either. You need an array with dtype=NAint32 or something (exact syntax
to be determined). Otherwise, they act exactly like any other values. In
particular, you can apply arithmetic functions and so forth to them. By
default, any function which takes an NA as an argument always returns an NA
as well, regardless of the values of the other arguments. This ensures that
if we try to compute the correlation of income with age, we will get "NA",
meaning "given that some of the entries could be anything, the answer could
be anything as well". This reminds us to spend a moment thinking about how we
should rephrase our question to be more meaningful. And as a convenience for
those times when you do decide that you just want the correlation between the
known ages and income, then you can enable this behavior by adding a single
argument to your function call.

For floating point computations, NAs and NaNs have (almost?) identical
behavior. But they represent different things -- NaN an invalid computation
like 0/0, NA a value that is not available -- and distinguishing between
these things is useful because in some situations they should be treated
differently. (For example, an imputation procedure should replace NAs with
imputed values, but probably should leave NaNs alone.) And anyway, we can't
use NaNs for integers, or strings, or booleans, so we need NA anyway, and
once we have NA support for all these types, we might as well support it for
floating point too for consistency.

A masked array is, conceptually, an ordinary rectangular numpy array, which
has had an arbitrarily-shaped mask placed over it. The result is,
essentially, a non-rectangular view of a rectangular array. In principle,
anything you can accomplish with a masked array could also be accomplished by
explicitly keeping a regular array and a boolean mask array and using numpy
indexing to combine them for each operation, but combining them into a single
structure is much more convenient when you need to perform complex operations
on the masked view of an array, while still being able to manipulate the mask
in the usual ways. Therefore, masks are preserved through indexing, and
functions generally treat masked-out values as if they were not even part of
the array in the first place. (Maybe this is a good heuristic: a length-4
array in which the last value has been masked out behaves just like an
ordinary length-3 array, so long as you don't change the mask.) Except, of
course, that you are free to manipulate the mask in arbitrary ways whenever
you like; it's just a standard numpy array.

There are some simple situations where one could use either of these tools to
get the job done -- or other tools entirely, like using designated surrogate
values (age=0), separate mask arrays, etc. But missing values are designed to
be particularly helpful in situations where the missingness is an intrinsic
feature of the data -- where there's a specific value that **should** exist,
if it did exist we'd it'd mean something specific, but it **doesn't**. Masked
arrays are designed to be particularly helpful in situations where we just
want to temporarily ignore some data that does exist, or generally when we
need to work with data that has a non-rectangular shape (e.g., if you make
some measurement at each point on a grid laid over a circular agar dish, then
the points that fall outside the dish aren't missing measurements, they're
just meaningless).

Initialization
^^^^^^^^^^^^^^

First, missing values can be set and be displayed as ``np.NA, NA``::

   >>> np.array([1.0, 2.0, np.NA, 7.0], dtype='NA[f8]')
   array([1., 2., NA, 7.], dtype='NA[<f8]')

As the initialization is not ambiguous, this can be written without the NA
dtype::

   >>> np.array([1.0, 2.0, np.NA, 7.0])
   array([1., 2., NA, 7.], dtype='NA[<f8]')

Masked values can be set and be displayed as ``np.IGNORE, IGNORE``::

   >>> np.array([1.0, 2.0, np.IGNORE, 7.0], masked=True)
   array([1., 2., IGNORE, 7.], masked=True)

As the initialization is not ambiguous, this can be written without
``masked=True``::

   >>> np.array([1.0, 2.0, np.IGNORE, 7.0])
   array([1., 2., IGNORE, 7.], masked=True)

Ufuncs
^^^^^^

By default, NA values propagate::

   >>> na_arr = np.array([1.0, 2.0, np.NA, 7.0])
   >>> np.sum(na_arr)
   NA('float64')

unless the ``skipna`` flag is set::

   >>> np.sum(na_arr, skipna=True)
   10.0

By default, masking does not propagate::

   >>> masked_arr = np.array([1.0, 2.0, np.IGNORE, 7.0])
   >>> np.sum(masked_arr)
   10.0

unless the ``propmask`` flag is set::

   >>> np.sum(masked_arr, propmask=True)
   IGNORE

An array can be masked, and contain NA values::

   >>> both_arr = np.array([1.0, 2.0, np.IGNORE, np.NA, 7.0])

In the default case, the behavior is obvious::

   >>> np.sum(both_arr)
   NA('float64')

It's also obvious what to do with ``skipna=True``::

   >>> np.sum(both_arr, skipna=True)
   10.0
   >>> np.sum(both_arr, skipna=True, propmask=True)
   IGNORE

To break the tie between NA and MSK, NAs propagate harder::

   >>> np.sum(both_arr, propmask=True)
   NA('float64')

Assignment
^^^^^^^^^^

is obvious in the NA case::

   >>> arr = np.array([1.0, 2.0, 7.0])
   >>> arr[2] = np.NA
   TypeError('dtype does not support NA')
   >>> na_arr = np.array([1.0, 2.0, 7.0], dtype='NA[f8]')
   >>> na_arr[2] = np.NA
   >>> na_arr
   array([1., 2., NA], dtype='NA[<f8]')

Direct assignnent in the masked case is magic and confusing, and so happens only
via the mask::

   >>> masked_array = np.array([1.0, 2.0, 7.0], masked=True)
   >>> masked_arr[2] = np.NA
   TypeError('dtype does not support NA')
   >>> masked_arr[2] = np.IGNORE
   TypeError('float() argument must be a string or a number')
   >>> masked_arr.visible[2] = False
   >>> masked_arr
   array([1., 2., IGNORE], masked=True)


Copyright
---------

This document has been placed in the public domain.
