Request to data source is outdated. Results in KeyError: 'flightSummaryList'`

[mitscay]

I have used the following on my script with Anaconda, since April but since 24/06/2020 whenever I try to run the library it returns the following:

`---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
in
1 # Save Arrivals CSV
----> 2 FlightTables.arrivals_csv("2020-07-01")
3
4 # Save Departures CSV
5 FlightTables.departures_csv("2020-07-01")

~\Anaconda3\lib\site-packages\flight_tables\flight_tables.py in arrivals_csv(self, iso_date_str)
10 @classmethod
11 def arrivals_csv(self, iso_date_str):
---> 12 self.date_to_csv(iso_date_str, "arrivals")
13
14 @classmethod

~\Anaconda3\lib\site-packages\flight_tables\flight_tables.py in date_to_csv(iso_date_str, direction)
30 heathrow_raw_dict = fetch_heathrow_data(iso_date_str, direction)
31
---> 32 batch_info = extract_batch_heathrow(heathrow_raw_dict)
33
34 parsed_flights = ParsedFlights(batch_info)

~\Anaconda3\lib\site-packages\flight_tables\heathrow_parsing.py in extract_batch_heathrow(response)
113 batch_info = [] # List with flight_info from multiple flights
114
--> 115 raw_flights = response['flightSummaryList']['flight']
116
117 for flight in raw_flights:
KeyError: 'flightSummaryList'`

I am not sure how to resolve this though.

[FergM]

Hi, thanks for raising the issue.

tldr: The data source has changed. The get request now needs a header.

Quick Fix:

• Update line 35 of heathrow_parsing.py as follows
• Old line:
  r = requests.get(url_to_scrape)
• New lines:

  headers={"origin": "https://www.heathrow.com"}
  r = requests.get(url_to_scrape, headers = headers)
  

For context:

• Line 30 in flight_tables.py is unsuccessful fetching the data.
• Ideally there should be error handling to catch this event more cleanly.
• I'm not sure when I'll get around to fixing. Ideally will add tests, error catching and an update to PyPi package.

Let me know if you have any other feedback about the documentation or context about your use case.

All the best

[mitscay]

Hi there, thank you for your quick feedback. As I am only a moderate user I am not quite certain how to go ahead and modify your script from my notebook? I have tried running the different scripts flight_tables.py, heathrow_parsing.py etc and I substituted the line you mentioned, however I keep receiving the same error message.

[FergM]

I've saved a fix to the develop branch for a start.

It should work if you save the develop branch locally and do command line execution (see README.md).

I'm planning to pull to master and update PyPi. Once that's done pip upgrade will work.

[mitscay]

Thanks @FergM for your help.

[FergM]

December 2020 Update:

• Heathrow updated the website a second time, so the first fix no longer works.
• The website is now much harder to scrape from.
• Pull requests welcome, particularly with parsing for other airports (which might be easier to access).