Prep for Analysis: Getting to know the CBC and NOAA Data

[rectheworld]

Prep for Analysis: Getting to know the CBC and NOAA Data

Goal:

Preform task required for analysis and compute tables and counts to help us understand the data better.

Inputs:

cbc_cleaned_usa_merged.csv - Contain volenteer submitted cbc data and the weather data for the closest noaa station. Distances are measured in meters.

Outputs:

A Notebook file containing the tasks mentioned

TASKS

• All temperature values should be divided by 10 to be comparable with the CBC readings (noaa station measures in tenth of a degree). These values should be kept in a New field til QA is done
• Create a variable to represent the existence of any PREC recorded by the cbc

Counting:

• Question: Getting to know the data by numbers. How many records are there? How many States? How many years?
• Question: How many records have cbc recorded TEMP, or PREC data but do not have the corresponding NOAA data?

Basic Stats and Investigations

• Take Absolute value of temp difference
• Plot over time to look at possible temperature difference
• Mean Difference
  For Temperature
• Outlier Check - who is way off, how is way off

[ijd5004]

I started doing some of this stuff already. I will formalize it into a notebook and post it before the datajam this Saturday.

[ijd5004]

Some of the temperature data has a negative sign in front of it. Do we know why?

[rectheworld]

Does it make sense for those to be a negative temperature for example -10°F?

…

Sent from my iPod Shuffle

On Feb 6, 2020, at 9:50 PM, ijd5004 ***@***.***> wrote: Some of the temperature data has a negative sign in front of it. Do we know why? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[ijd5004]

Very possible. I will look up the locations of a few places to see if it passes the sanity check (i.e. no negatives in florida)

[ijd5004]

Looking at some of the negative temperatures in Florida. Tried to verify with a manual pull of data from a Gainesville station from the NOAA website. The temps do not match. Are we confident the NOAA temps are in a tenth of a degree F. -0.6 F is 30.92 C...so that would match if the NOAA data from Big Query was Celcius.

GHCND_USW00012816_1989-12-1.pdf

[Frankie-Figz]

NOAA temps are in tenth of a degree Celcius. Negative Farenheit temperature in Florida sounds highly dubious

…

On Fri, Feb 7, 2020, 8:13 PM ijd5004 ***@***.***> wrote: Looking at some of the negative temperatures in Florida. Tried to verify with a manual pull of data from a Gainesville station from the NOAA website. The temps do not match. Are we confident the NOAA temps are in a tenth of a degree F. -0.6 F is 30.92 C...so that would match if the NOAA data from Big Query was Celcius. [image: image] <https://user-images.githubusercontent.com/30261267/74076429-fed60a00-49e5-11ea-8bf7-8f39605ebff7.png> GHCND_USW00012816_1989-12-1.pdf <https://github.com/DataKind-DC/audubon-cbc/files/4173573/GHCND_USW00012816_1989-12-1.pdf> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=AJQJN6QTUNLVSDYEZRKFMVTRBYBMRA5CNFSM4KP2OZK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFEI4A#issuecomment-583681136>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJQJN6WNJXZDMTTIGW7PKBDRBYBMRANCNFSM4KP2OZKQ> .

[ijd5004]

Dubious indeed. Celcius makes much more sense.

[ijd5004]

Tasks completed:

• NOAA temperature conversion
• Logical variable for precipitation
• Take Absolute value of temp difference
• Mean Difference for Temperature
• Outlier Check - using z-scores of temperature difference between circle and station
• Counting: how many rows, how many rows where distance under 10km, how many z-score outliers, etc.

(I'm not convinced my outlier analysis is useful yet. More important is probably determining the distances, altitudes, and other measurements cutoff values between the circles and stations.)

[ijd5004]

I believe I have answered all the questions in the prompt for this issue. We will definitely need a deep dive on outliers and visualizations, though.