Using openweathermap api – part 2

In part 1 we learned how to use endpoint to read current weather and forecast in browser and now lets make a python package.

But just before lets make few points.

  • Our reader should work in background as thread,
  • Have ability to read data for few cities
  • Read data not in real time but with some frequency

Why ?
Simple, we don’t want to send requests to openweathermap server with each call. It would be lame. Lets imagine that main loop is reading and displaying weather each second (on nice 20×4 LCD). Doing each second a call to remote server would take ages (ok few ms but way to long ūüôā ). We are asking for data and we want it now. And what is also important data wont change so often. We don’t need to have a fresh packet from server.

So this is part where threading comes to play. It will read data from weather endpoint each 10 minutes and from forecast endpoint each hour. Next it will parse data to our format and store it. So when we ask our package for weather date it will return it from cache.

Setup

So hopefully you have some experience in python programming, I wont dig into basic things. We will be using Python 2.7 on Raspberry Pi. But coding is on windows machine on mapped drive. Home directory on Raspi is mapped to drive S: on my main computer.

There will be some functions or structures that are not necessary in this tutorial but necessary in RaspiCmd. So I just left them and reused. Such thing is worker.py file. We can live without it and move all functions to other file. Another thing are start() and stop() functions. I will try and point out such places.

IDE that I’m using is PyCharm. Setting remote interpreter to Raspi allows us to have a full code completion.

Start your favourite IDE (PyCharm :)), create new project (openweather) and new package (weather). Now create some empty files: weather/__init__.py,  weather/openweather.py, weather/weather_codes.py, weather/worker.py and weather_intro.py.

At the end we should have a following structure:

openweathermap_files_structure

Weather codes

As I wrote earlier (part 1) I’m using my own dictionary to translate weather code to description. Its not necessary but I need as short description as possible and readable. So open file weather_codes.py and copy:

#!/usr/bin/python
# -*- coding: utf-8 -*-
__author__ = 'kosci'

weather_states = {
    200: "TR-",  # 'thunderstorm with light rain',
    201: "TR",  #'thunderstorm with rain',
    202: "tr+",  #'thunderstorm with heavy rain',
    210: "t-",  #'light thunderstorm',
    211: "t",  #'thunderstorm',
    212: "t+",  #'heavy thunderstorm',
    221: "t!",  #'ragged thunderstorm',
    230: 'thunderstorm with light drizzle',
    231: 'thunderstorm with drizzle',
    232: 'thunderstorm with heavy drizzle',

    300: 'D-', #'light intensity drizzle',
    301: 'drizzle',
    302: 'heavy intensity drizzle',
    310: 'light intensity drizzle rain',
    311: 'drizzle rain',
    312: 'heavy intensity drizzle rain',
    313: 'shower rain and drizzle',
    314: 'heavy shower rain and drizzle',
    321: 'shower drizzle',

    500: "R-",  #'light rain',
    501: "R",  #'moderate rain',
    502: "R+",  #'heavy intensity rain',
    503: "R++",  #'very heavy rain',
    504: "R!",  #'extreme rain',
    511: 'freezing rain',
    520: 'light intensity shower rain',
    521: 'shower rain',
    522: 'heavy intensity shower rain',
    531: 'ragged shower rain',

    600: 'S-', #light snow',
    601: 'S', #snow',
    602: 'S+', #heavy snow',
    611: 'sleet',
    612: 'shower sleet',
    615: 'light rain and snow ',
    616: 'rain and snow ',
    620: 'light shower snow',
    621: 'shower snow',
    622: 'heavy shower snow',

    701: 'mist',
    711: 'smoke',
    721: 'haze',
    731: 'Sand/Dust Whirls',
    741: 'Fog',
    751: 'sand',
    761: 'dust',
    762: 'VOLCANIC ASH',
    771: 'SQUALLS',
    781: 'TORNADO ',

    800: "C0",  #'sky is clear',
    801: "C-",  #'few clouds',
    802: "C",  #'scattered clouds',
    803: "C+",  #'broken clouds',
    804: "C!",  #'overcast clouds',
}

If you need translate all codes to short form.

 Worker

My system is using worker file as controller for main package body. Function defined in it are callable from system. That’s why in this tutorial we will be using such file.

This worker is very simple and gives us four functions. Start and shutdown are required by system. For user are weather and forecast.

So open worker.py and copy:

#!/usr/bin/python
# -*- coding: utf-8 -*-
__author__ = "Kosci"
__version__ = "1.0"
from openweather import Weather


class Worker:
    def __init__(self, core):
        self.core = core
        self.init()
        self.handler = Weather(self.cities)

    def init(self):
        """basic setup"""
        self.cities = {
¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬† 3103402: 'Bielsko-BiaŇāa',
            2946447: 'Bonn'
        }

    def start(self):
        """start worker"""
        self.handler.start()

    def shutdown(self):
        """shudown worker"""
        self.handler.stop()

    def weather(self, city_id=None, key=None):
        """return curent weather"""
        if key is None:
            return self.handler.weather(city_id)
        return self.handler.weather(city_id)[key]

    def forecast(self, city_id=None, forecast_date=None):
        """return forecast"""
        return self.handler.forecast(city_id, forecast_date)

What’s going on here ?

We import class Weather that will have all functions, all algorithms and thread. It will do main job. We will take care of it later.

Next we define Worker class and constructor __init__(self, core). Variable core is not used in this tutorial so you may remove it or just pass None when creating class.
Most important thing is self.handler as it creates main class. In a parameter we send  dictionary with our cities.

In function init(self) we have defined cities to watch, two ids and city names.

Functions start(self) and shutdown(self) are required by system and I just left them and reused. They start and stop Weather thread that download JSON from endpoints and parse them.

Function weather(self, city_id=None, key=None) calls Weather class and retrieve current weather. We may specify city_id parameter or leave it as None. Setting to None will return data for first city in dictionary (first is arbitrary because there is no such thing as order in dictionary). Data is also in dictionary format with all weather description fields or if we supply key it will return one record for this key. Simplest call is just .weather()

Returned structure may look like this:

{'wind_speed': 2.22, 
 'pressure': 1015.6, 
 'temperature_current': 20.35, 
 'clouds': 0, 
 'weather': 'C0', 
 'wind_deg': 202.5, 
 'update': 1409308839, 
 'weather_id': 800, 
 'humidity': 33}

Last function is forecast(self, city_id=None, forecast_date=None). This function also calls Weather class and get weather forecast . AS in weather function we may send city_id or leave it as None. If we don’t send forecast_date next day will be assumed. Return from this function may look like this:

{'wind_speed': 1.71, 
 'pressure': 975.22, 
 'temperature_max': 22.8, 
 'weather': 'C0', 
 'clouds': 0, 
 'wind_deg': 93, 
 'temperature_min': 10.77, 
 'weather_id': 800, 
 'humidity': 79}

Class Weather and getting weather

Its time to look at main class Weather. For now open openweather.py file and copy

#!/usr/bin/python
# -*- coding: utf-8 -*-
__author__ = "Kosci"
__version__ = "1.1"

import time
import threading
import urllib2
import json
import datetime
import weather_codes
import socket


class Weather(threading.Thread):
    def __init__(self, cities):
        threading.Thread.__init__(self)
        self.cities = cities
        self.work = True
        self.tick = {'sleep': 10, 'weather_counter': 6*10, 'forecast_counter': 6*60}
        self.tick['_wcounter'] = self.tick['weather_counter'] + 1
        self.tick['_fcounter'] = self.tick['forecast_counter'] + 1

        self.weather_row_stub = {
            'update': 0,
            'temperature_min': 0,
            'temperature_max': 0,
            'temperature_current': 0,
            'humidity': 0,
            'pressure': 0,
            'wind_speed': 0,
            'wind_deg': 0,
            'clouds': 0,
            'weather_id': 0,
            'weather': ''
        }
        self.current_weather_raw = {i: "" for i in cities}
        self.current_weather = {i: self.weather_row_stub.copy() for i in cities}

        self.forecast_weather_raw = {i: {} for i in cities}
        self.forecast_weather = {i: {} for i in cities}

        self.url_current = "http://api.openweathermap.org/data/2.5/weather?id=%CITY_ID%&units=metric&mode=json"
        self.url_forecast = "http://api.openweathermap.org/data/2.5/forecast/daily?id=%CITY_ID%&mode=json&units=metric&cnt=4"

    def run(self):
        """main loop, reads data from openweather server"""
        while self.work:
            if self.tick['_wcounter'] > self.tick['weather_counter']:
                for city_id in self.cities:
                    """current weather"""
                    url = self.url_current.replace("%CITY_ID%", str(city_id))
                    json_data = self._fetch_data(url)
                    if json_data:
                        self.current_weather_raw[city_id] = json_data
                        self._decode(city_id)
                        self.tick['_wcounter'] = 0

            if self.tick['_fcounter'] > self.tick['forecast_counter']:
                for city_id in self.cities:
                    """ forecast """
                    url = self.url_forecast.replace("%CITY_ID%", str(city_id))
                    json_data = self._fetch_data(url)
                    if json_data:
                        for row in json_data['list']:
                            date = (datetime.datetime.fromtimestamp(int(row['dt']))).strftime("%Y-%m-%d")
                            self.forecast_weather_raw[city_id][date] = row
                            self.forecast_weather[city_id][date] = self._decode_forecast(row)
                        self.tick['_fcounter'] = 0

            time.sleep(self.tick['sleep'])
            self.tick['_wcounter'] += 1
            self.tick['_fcounter'] += 1

    def stop(self):
        self.work = False

    def get_raw_data(self, city_id=None):
        """return raw weather data"""
        if city_id is None:
            return self.current_weather_raw.itervalues().next()
        else:
            return self.current_weather_raw[city_id]

    def weather(self, city_id=None):
        """return decoded weather"""
        if not city_id:
            city_id = self.cities.keys()[0]

        return self.current_weather[city_id]

    def forecast(self, city_id=None, date=None):
        """return forecast"""
        if not city_id:
            city_id = self.cities.keys()[0]
        if date is None:
            date = time.strftime("%Y-%m-%d", time.localtime(time.time() + 24*3600))

        if not date in self.forecast_weather[city_id]:
            return self.weather_row_stub

        return self.forecast_weather[city_id][date]

    def _fetch_data(self, url):
        """fetch json data from server"""
        try:
            request = urllib2.Request(url, None, {'User-Agent': 'Raspberry PI / rpicmd'})
            response = urllib2.urlopen(request)
            data = response.read()
            json_data = json.loads(data)
        except urllib2.URLError as e:
            json_data = None
            print time.strftime("%Y-%m-%d %H:%M:%S"), "error fetching from url", url, "\nreason", e.reason
        except socket.timeout as e:
            json_data = None
            print time.strftime("%Y-%m-%d %H:%M:%S"), "time out error from url", url, "\nreason", e.reason
        except ValueError as e:
            json_data = None
            print "Decode failed"

        return json_data

    def _decode(self, city_id):
        self.current_weather[city_id] = {
            'temperature_current': self.current_weather_raw[city_id]['main']['temp'],
            'humidity': self.current_weather_raw[city_id]['main']['humidity'],
            'pressure': self.current_weather_raw[city_id]['main']['pressure'],
            'wind_speed': self.current_weather_raw[city_id]['wind']['speed'],
            'wind_deg': self.current_weather_raw[city_id]['wind']['deg'],
            'weather_id': self.current_weather_raw[city_id]['weather'][0]['id'],
            'weather': self._weather_codes(self.current_weather_raw[city_id]['weather'][0]['id']),
            'clouds': self.current_weather_raw[city_id]['clouds']['all'],
            'update': self.current_weather_raw[city_id]['dt']
        }

    def _decode_forecast(self, row):
        return {
            'temperature_min': row['temp']['min'],
            'temperature_max': row['temp']['max'],
            'humidity': row['humidity'],
            'pressure': row['pressure'],
            'wind_speed': row['speed'],
            'wind_deg': row['deg'],
            'weather_id': row['weather'][0]['id'],
            'weather': self._weather_codes(row['weather'][0]['id']),
            'clouds': row['clouds']
        }

    def _weather_codes(self, id):
        """return weather description"""
        return weather_codes.weather_states[id]


Class Weather extends Thread.

Lets start from __init__(self, cities).  We set basic variables and define endpoint urls. First is self.tick dictionary. Its made that way to avoid common problem.

When thread enter time.sleep(n) it will stuck on it. So when sleeping for 10 minutes  you press ctrl+c to abort program it wont happen. Thread will be waiting for sleep to end and your program will hangs. So to minimize sleep gap thread is sleeping in small intervals 10 seconds. And each tick increments counters (_wcounter and _fcounter) and when they go over limit (weather_counter, forecast_counter) main functions are called.

Variable weather_row_stub define how our return record looks alike.

Endpoint urls are stored in url_current and url_forecast. If you look at the end of url_forecast you will see number 4. It define for how many days we want a forecast.

Function run(self) is responsible for doing two calls to openweathermap.
Lets see how it works. In while are two ifs, sleep and counters. So each while loop do job and sleep for self.tick[‘sleep’] – 10s by defaults and increase counters. What is important sleeps takes 10s but ifs are activated according to defined thresholds.

In while self.work are two ifs. They activate when corresponding counter reaches threshold. First (_wcounter) activate current weather fetch and second calls weather forecast endpoint.

Current weather is fetched for each city_id passed to constructor. URL is created and passed to self._fetch_data(url). This function do actual url call and tries to convert received data to JSON format. If proper JSON is returned data is stored and decoded according to self.weather_row_stub. This is done in function self._decode(self, city_id)

Forecast is done almost same way. Call is made, decoded to JSON and stored.

Function stop(self) is simple, just sets self.work to false. So while in run will stop

Two functions are responsible for returning data to user. They are called from worker.py. Its weather(self, city_id=None) and forecast(self, city_id=None, date=None).

Last important function is _fetch_data(self, url). It builds urllib request and read data from endpoint. Received data is parsed to JSON format. If any error occur function return None.

Functions that decode received JSON are just simple assignment to our array.

 Lets run it !

Its time to put it all together and run. Open weather_intro.py and copy:

#!/usr/bin/python
# -*- coding: utf-8 -*-
__author__ = 'kosci'
from weather import worker
import time

if __name__ == '__main__':
    w = worker.Worker(None)
    print "start"
    w.start()
    try:
        while 1:
            d = w.weather()
            print "current weather :", d
            d = w.forecast()
            print "forecast :", d
            print "-----------------"
            time.sleep(5)
    except KeyboardInterrupt:
        print "closing..."
    except:
        raise
    finally:
        w.shutdown()

Now run this script and you will see some weather data. In first or even second loop you will  probably see empty values Рdata is not received from server. But after third you should see two records with full data.

current weather : {'wind_speed': 0, 'clouds': 0, 'update': 0, 'weather_id': 0, 'humidity': 0, 'pressure': 0, 'temperature_max': 0, 'weather': '', 'wind_deg': 0, 'temperature_min': 0, 'temperature_current': 0}
forecast : {'update': 0, 'pressure': 0, 'weather': '', 'humidity': 0, 'wind_speed': 0, 'clouds': 0, 'weather_id': 0, 'wind_deg': 0, 'temperature_max': 0, 'temperature_current': 0, 'temperature_min': 0}
-----------------
current weather : {'wind_speed': 5.81, 'pressure': 1016.5, 'temperature_current': 20.73, 'clouds': 0, 'weather': 'C0', 'wind_deg': 67.5, 'update': 1409847837, 'weather_id': 800, 'humidity': 63}
forecast : {'update': 0, 'pressure': 0, 'weather': '', 'humidity': 0, 'wind_speed': 0, 'clouds': 0, 'weather_id': 0, 'wind_deg': 0, 'temperature_max': 0, 'temperature_current': 0, 'temperature_min': 0}
-----------------
current weather : {'wind_speed': 5.81, 'pressure': 1016.5, 'temperature_current': 20.73, 'clouds': 0, 'weather': 'C0', 'wind_deg': 67.5, 'update': 1409847837, 'weather_id': 800, 'humidity': 63}
forecast : {'wind_speed': 1.71, 'pressure': 975.22, 'temperature_max': 22.8, 'weather': 'C0', 'clouds': 0, 'wind_deg': 93, 'temperature_min': 10.77, 'weather_id': 800, 'humidity': 79}
-----------------

And that is all. Have fun playing and optimizing code.

Advertisements

One comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s