Missing proper motion values in AGASC

[taldcroft]

Just randomly looking at star information from a query at ra=10, dec=20, I see a few stars that have a defined proper motion in one axis (RA) but not the other. This doesn't make any sense. This is actually in the original FITS catalog, where the RA and DEC proper motion cols are the 7th and 8th, and have values 11 and -9999 from catid=5 (Tycho-2).

ascds-fido% mp_get_agascid 155980472
AGASC_INPUT         =
  /data/agasc1p6/agasc/
AGASC_INPUT_FILEMAP =
  /data/agasc1p6/tables/outfilemap
 155980472  10.3686299  18.9845742    33 5 2000.0000    11 -9999 5    22    24 3  10.21719     8     0  10.04000     5     4 5   0.29665     6 5   0.34900     7 5 -9999.00000 -9999 0 -9999 0   0  0 364   144   144   144   144   144   144       599    116693       599         1     -9999 -9999 -9999 -9999

This star definitely has a dec proper motion from Tycho:
http://vizier.u-strasbg.fr/viz-bin/VizieR-S?TYC%201190-599-1

If @malgosias is making an AGASC 1.7 this should get addressed.

[jeanconn]

I thought we'd somewhat decided that first round would be a MAG_ACA fix of some kind and then we'd go and try to fix position related things if we thought there'd be any value in that. Sounds like there would be value in that.

I've yet to come across the original AGASC code to see how PM_RA and PM_DEC were assigned from source catalogs.

[jeanconn]

And it doesn't seem like this is an agasc module issue but I agree there isn't really a good/better place to keep this.

[taldcroft]

Releasing a new AGASC is a huge deal, so if there is real value added for low-hanging fruit we should do it now. In this case it should be straightforward to select out all cases of a half-missing proper motion for a star from Tycho-2 and fix it. In basically the first random example I tried there was a 8.9 mag star that would get rejected. So for now likely have to work around this bug in the AGASC somehow.

I've yet to come across the original AGASC code to see how PM_RA and PM_DEC were assigned from source catalogs.

And I hope you've accepted that that the code is probably lost. But for Tycho-2 is there any ambiguity? At least the one example I looked just shows that it converted 10.9 mas/yr to 11, as expected.

[jeanconn]

Releasing a new AGASC is a huge deal, so if there is real value added for low-hanging fruit we should do it now.

Agreed. Was just still working from

https://github.com/sot/ska-projects/issues/81#issuecomment-379000688

And I hope you've accepted that that the code is probably lost

Not really. I think proper motion didn't get in there until AGASC 1.5, so it isn't that old.

But for Tycho-2 is there any ambiguity?

Don't know. If the original code had something like "don't bother to paste in the value if below N or if the error on the value is greater than N" or something we'd have learned something here. I realize sometimes the why doesn't matter but I obviously prefer things wrapped up with a bow.

[taldcroft]

AGASC version 1.5 March 2002. That's 16 years and a few staff changes ago. I actually looked at the FDB promotion presentation (by Paul Levitt) and it didn't haven't much of use, basically just showing interface changes.

The comment you referenced was about potentially improving valid Tycho values in the AGASC, not correcting obviously incorrect (-9999) values.

[jeanconn]

The comment you referenced was about potentially improving valid Tycho values in the AGASC, not correcting obviously incorrect (-9999) values.

Gotcha. Yeah, my take-away had just been that other fixes were orthogonal to what @malgosias was doing now. For these PM fixes, I suppose instead of trying to improve upon the positions or proper motion values we just want to lookup the Tycho 2 values for the stars in the AGASC that seem to be missing. And we'd probably do that just via the Tycho 2 ids that are already in AGASC 1.6.

[jeanconn]

It looks like this came up on the observation I'm trying to V&V

In [29]: slot4 = agasc.get_star(77990824)
In [30]: slot4['PM_DEC']
Out[30]: -9999
In [31]: slot4['PM_RA']
Out[31]: 119

And it looks like the pipeline doesn't do any one-axis proper motion fixes:

        if ( ( fabs(middate) < FLT_EPSILON ) ||
             ( fabs( -9999.0 - epoch ) < FLT_EPSILON ) ||
             ( fabs( -9999.0 - gsprops.pm_ra )  < FLT_EPSILON ) ||
             ( fabs( -9999.0 - gsprops.pm_dec ) < FLT_EPSILON ) )
	{
          // Skip it..  

[jeanconn]

Though it also looks like this has a PM_DEC of 2.4 in Tycho-2 and I suppose either of our integer choices (2 or 3) would have had significant relative error.

[taldcroft]

would have had significant relative error.

Sure, but absolute error is what matters. But anyway I made some progress here:

stars = agasc.get_agasc_cone(10, 20, 20)
sok = stars[(stars['POS_CATID'] == 5) & (np.abs(stars['PM_RA']) > 100) & (stars['PM_DEC'] == -9999) & (stars['PM_RA'] != -9999)]
sok['PM_RA', 'PM_DEC', 'POS_CATID', 'MAG_ACA', 'RA', 'DEC']

Then I picked assorted values and checked with SIMBAD. In every case (maybe 10 tries) the proper motion with -9999 was either less than 6 mas/yr or less than 3-sigma from 0. E.g.
http://simbad.u-strasbg.fr/simbad/sim-coo?Coord=352.968904273+19.93715277&Radius=2&Radius.unit=arcsec&submit=submit+query

Note in this one that the SIMBAD PM is 17.3 mas/yr => 0.34 arcsec at 20 years. Not huge but noticeable.

Takeaway is that we should:

• Do up a formal notebook with hundreds or thousands of queries to confirm this anecdotal evidence. I only looked at Tycho2 positions.
• Treat the two axes (RA, Dec) separately in all ground software. A value of -9999 just means use 0.0 for PM in that axis. I note that the Python agasc module is already doing this, even though formally "don't do any correction because we don't have good data" is different from "the correction is close to zero". But it turns out the latter is correct.
• Put in an ECR on the aspect pipeline.

Also, -9999 is quite common for stars brighter than 10.6 with class=0:

In [41]: sok['PM_RA'                                                   
, 'PM_DEC', 'POS_CATID', 'MAG_ACA', 'RA', 'DEC']
Out[41]: 
<Table masked=False length=13624>
PM_RA PM_DEC POS_CATID MAG_ACA      RA         DEC    
int16 int16    uint8   float32   float64     float64  
----- ------ --------- ------- ----------- -----------
-9999    -17         5 9.69345  8.69698066   0.0972845
  -17    -39         5 9.55616 10.42911084   0.1123006
-9999  -9999         6  10.496     9.69655     0.18806
  -13    -10         5 10.4841 11.90229811  0.19007102
-9999     -7         5 10.0743  9.61172531  0.20971404
  265    -52         5 7.09766 11.29566353  0.25328207
-9999    -25         5 9.80681 12.44533169  0.25947476
  -10    -14         5 10.3269 12.32517477  0.26663648
   -8  -9999         5 10.5856 12.85965786  0.26766761
-9999    -17         5 9.31052 12.60042287  0.31616043
  -11    -10         5  9.9435 13.47542314  0.33346735
-9999  -9999         6  10.556    12.26804     0.34496
   63    -26         5 8.19333 12.13842488  0.38204835
   47     18         5 10.2723  13.1233768  0.43544931
 -137    -86         5 8.40284  6.51266962  0.43755966
-9999    -26         5 9.45337  9.48819274    0.437625

So it's looking like millions of cases if we fix this. But maybe we don't have to if we understand that -9999 is just a proxy for "small enough".

[jeanconn]

Sure, but absolute error is what matters.

Right. I was just noting that I can see where this could have happened.

Regarding:

Do up a formal notebook with hundreds or thousands of queries to confirm this anecdotal evidence

Do you mean this along the lines of just confirming they are all small and then treating them like 0 instead of getting a new value for the missing data? It looks like there are ~700 thousand stars with the mismatch (defined PM in one axis in the miniagasc), but I haven't confirmed how fast it would be to just get an estimate for the missing axis for all of them.

[taldcroft]

Sigh RT*D... So anything less than 4-sigma is just dropped. I must not have been paying attention at the time.

------------------------------------------------------
ASSIGNING AGASC1.5 VALUES FROM Tycho-2 catalog.dat
------------------------------------------------------
It's virtually always true that TYC2 positions and proper motions
are best, so we include them without comparison to existing AGASC
position errors.  AGASC has no proper motion error column,
so we first test for the significance of proper motion before
updating.  There seem to be multiple position entries (one for each
observation by Tycho). We should use the MEAN value.

	pflag	  N	  Meaning
	-----	-------	------------------------------
	BLANK	2417370	normal mean position and p.m.
	X	109445	no mean position or p.m.
	P	13098	mean position from 2 entries (close dbl star?)

------------------
AGASC_ID
------------------
   AGASC_ID= 131072*TYC1 + 8*TYC2 + (TYC3-1)

------------------
RA, DEC and PM
------------------
Tycho-2 position and PM values are more trustworthy than
any previous values.  For every star appearing in Tycho-2, we
will overwrite existing values from any previous catalog (ACT,
Tycho-1, PPM, GSC1.1) with either the Tycho-2 value or the default.

  spmra=0
  spmdec=0
  pmthresh=4
  PM_RA=-9999
  PM_DEC=-9999
  PM_CATID=0
  if(pflag .eq. BLANK .OR. pflag .eq. 'P')then
    RA=		mRAdeg
    DEC=	mDEdeg
    POS_ERR=	sqrt(e_mRA**2 + e_mDE**2)
    POS_CATID=	5
    EPOCH=	2000.0
    if( (abs(pmRA)/e_pmRA).gt.pmthresh)then
	PM_RA=pmRA
	PM_CATID=5
	spmra=	pmRA
    endif
    if( (abs(pmDE)/e_pmDE) .gt.pmthresh)then
	PM_DEC=	pmDE
	PM_CATID=5
	spmdec=	pmDE
    endif

c   Really large p.m. stars are flagged as ASPQ2=1
c   so calculate squared sum for comparison to high
c   p.m. threshold of 500mas/yr for flag.

    ASPQ2=0
    spmtot= sqrt(spmra**2 + spmdec**2)
    if(spmtot.ge. 500)	ASPQ2=1

c Note that in the following condition, any existing AGASC1.4 
c PM_RA, PM_DEC, and PM_CATID should be set to the default null values.
  if(pflag .eq. 'X')then
    RA=		RAdeg
    DEC=	DEdeg
    POS_ERR=	sqrt(e_RA**2 + e_DE**2)
    POS_CATID=	5
    EPOCH=	0.5(epRA + epDE)
  endif

[jeanconn]

Ah. So now I've learned about https://icxc.harvard.edu/mp/ascii/ which might be useful.

[jeanconn]

But I think the biggest issue is still that pipeline bit of code that skips any star with either set to -9999.

[jeanconn]

I don't know if I'm noticing this more in V&V just because we don't have margin to discard stars, but I've got another observation today that could use this fix. That makes 3 in 2 weeks. Do we still want to wait for AGASC 1.7 or ask if this could go in a DS patch sooner?

[taldcroft]

DS patch sooner.