Subject: [Geojson] Question regarding RFC 7946

From: "V. Kelly Bellis" <[email-masked]>
Date: Sat, 16 Mar 2019 12:51:32 -0400

Hello [email-masked]

Trying to get OpenWeatherMap's list cities into the GeoJSON format has 
been a great excuse to get reacquainted with regex; however (after two 
days of failures), I'm wondering: is there a better way?

OpenWeatherMap.org <https://openweathermap.org/> uses a list of cities 
by ID: http://bulk.openweathermap.org/sample/city.list.json.gz

Typical format contained in city.list.json

I first thought a typical record looks similar to the format in GeoJSON 
<http://geojson.org/>, but not similar enough for Global Mapper which 
can read GeoJSON files. Here is the result after attempting to read the 
original OWM format:

After starting with the easy change of finding "coords" and changing 
to"coordinates", I eventually developed:

Find: ^  {\n    "id": ([0-9]*),\n    "name": 
"([A-Za-z\u00C0-\u017F&\s\'\‘\’]*)",\n    "country": "([A-Z]*)",\n    
"coordinates": {\n      "lon": ([-]*)([0-9]*.[0-9]*),\n      "lat": 
([-]*)([0-9]*.[0-9]*)\n    }\n  },\n

Replace with: {\n  "type": "Feature",\n  "geometry": {\n    "type": 
"Point",\n "coordinates": [\4\5, \6\7]\n  },\n  "properties": {\n 
"city_name": "\2"\n    "country_code": "\3"\n    "owm_id": "\1"\n  }\n},\n

 From what I've gathered, it is closer to being in the (attached) 
geojson format (though I haven't dared try and parse 
https://tools.ietf.org/html/rfc7946 as it's a bit intimidating), but 
still it fails. And what little I read of it, GeoJSON generally appears 
to be unforgiving and laden with rigidities.

I've been testing using only regex and a small sample of OpenWeatherMap 
records wherein I added algebraic signs; i.e., tossed in a couple minus 
symbols for lon and lat, and added some odd characters, found naturally 
in the larger dataset.

Here's a before and after screen:

Still, Global Mapper balked at my reformatting:

At this point, I'm not sure what else I can do to get GM to read the OWM 
stuff. And to be clear, this was more of an educational lark than 
anything, all stemming from simply wanting to know the OWM city IDs in a 
given area.

FWIW, I also wrote to [email-masked] asking:

 1. Why isn't OWM's format the same as the standard GeoJSON?
 2. And in followup, could you please consider offering the city id list
    in GeoJSON format in addition to your own OWM format?

Also, I'm in close contact with Blue Marble Geographics.

Maybe I'll hear from them next week..

Thank you! :)

Kind regards,

Kelly

On 3/15/2019 11:29 AM, Mike Childs wrote:
> Hi Kelly,
>
> Their .json file is indeed slightly different than the standard 
> GeoJSON format. I'm not sure whey they chose to get so close to the 
> actual standard but then modify a few things.
>
> Our GeoJSON import capabilities come from a 3rd-party library (GDAL), 
> so we'd need them to add support for this flavor, or you could modify 
> the file directly in a text editor to match the expected format.
>
> I think you could start by changing "coords" to "coordinates", but 
> more may be needed.
>
> Best Regards,
>
> *Mike Childs* | Global Mapper Guru
> Blue Marble Geographics
> www.bluemarblegeo.com 
> <http://www.bluemarblegeo.com/?utm_source=signature&utm_medium=email&utm_campaign=website>
>
> 22 Carriage Lane
> Hallowell, ME 04347 USA
> /Phone:/ +1 (207) 622-4622
> /Toll Free:/ (800) 616-2725
> /Email:/ [email-masked] <mailto:[email-masked]>
>
> Please note, [email-masked] 
> <mailto:[email-masked]> is no longer an active email - 
> please update your tech support contact to [email-masked] 
> <mailto:[email-masked]>*.
>
> Blue Marble GeoTalks on March 21, 2019 
> <http://www.bluemarblegeo.com/bmuc/index.php?aid=siggeotalks01152019&utm_source=signature&utm_medium=email&utm_campaign=geotalks_2019&utm_term=http://www.bluemarblegeo.com/bmuc/index.php&utm_content=signature>
>
> 	Follow us: 	Facebook 
> <https://www.facebook.com/bluemarble.geographics> 	Twitter 
> <https://twitter.com/BlueMarbleGeo> 	LinkedIn 
> <http://www.linkedin.com/company/blue-marble-geographics> 	YouTube 
> <http://www.youtube.com/user/BlueMarbleWebinars> 	Instagram 
> <https://www.instagram.com/bluemarblegeo/> 	
>
> The information in this email message is legally privileged and 
> confidential information intended only for the use of the 
> individual(s) named above. If you, the reader of this message, are not 
> the intended recipient, you are hereby notified that you should not 
> further disseminate, distribute, or forward this email. If you have 
> received this email in error, please notify the sender and delete all 
> copies of the message promptly. Thank you.
>
>
>
> On Fri, Mar 15, 2019 at 8:38 AM V. Kelly Bellis <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     OpenWeatherMap.org <https://openweathermap.org/> uses a list of
>     cities by ID: http://bulk.openweathermap.org/sample/city.list.json.gz
>
>     Typical format contained in city.list.json
>
>     A typical record looks similar to the format in GeoJSON, but not
>     similar enough for Global Mapper.
>
>
>     Doubt there's much public need for this, and am just curious - Is
>     it too much bother to expand GM's format choices to include OWM's
>     json? I'll give TextPad and regex  a try just for fun! :)
>
>     Thank you! :)
>
>     Kind regards,
>
>     Kelly
>
>
>     -- 
>     V. Kelly Bellis, PLS
>     17 Union Street
>     Ellsworth, ME 04605
>
>     Precisely */Determining True North
>     <https://www.youtube.com/watch?v=zMwDkzl43n8>/* without a surveyor!
>
-- 
V. Kelly Bellis, PLS
17 Union Street
Ellsworth, ME 04605

Precisely */Determining True North 
<https://www.youtube.com/watch?v=zMwDkzl43n8>/* without a surveyor!

---

From: Martin Daly <[email-masked]>
Date: Mon, 18 Mar 2019 09:06:53 +0000

You have no “FeatureCollection” object, and therefore no “features” array. Your “properties” object does not have commas between consecutive key/value pairs. That’s two problems to be going on with.

You can use jsonlint.com and geojsonlint.com to check your output, which may give clearer errors than you have been getting.

Martin

From: GeoJSON <[email-masked]> On Behalf Of V. Kelly Bellis
Sent: 16 March 2019 16:52
To: [email-masked]
Subject: [Geojson] Question regarding RFC 7946

Hello [email-masked]<mailto:[email-masked]>

Trying to get OpenWeatherMap's list cities into the GeoJSON format has been a great excuse to get reacquainted with regex; however (after two days of failures), I'm wondering: is there a better way?

OpenWeatherMap.org<https://openweathermap.org/> uses a list of cities by ID: http://bulk.openweathermap.org/sample/city.list.json.gz

I first thought a typical record looks similar to the format in GeoJSON<http://geojson.org/>, but not similar enough for Global Mapper which can read GeoJSON files. Here is the result after attempting to read the original OWM format:

After starting with the easy change of finding "coords" and changing to "coordinates", I eventually developed:

Find: ^  {\n    "id": ([0-9]*),\n    "name": "([A-Za-z\u00C0-\u017F&\s\'\‘\’]*)",\n    "country": "([A-Z]*)",\n    "coordinates": {\n      "lon": ([-]*)([0-9]*.[0-9]*),\n      "lat": ([-]*)([0-9]*.[0-9]*)\n    }\n  },\n

Replace with: {\n  "type": "Feature",\n  "geometry": {\n    "type": "Point",\n    "coordinates": [\4\5, \6\7]\n  },\n  "properties": {\n    "city_name": "\2"\n    "country_code": "\3"\n    "owm_id": "\1"\n  }\n},\n

From what I've gathered, it is closer to being in the (attached) geojson format (though I haven't dared try and parse https://tools.ietf.org/html/rfc7946 as it's a bit intimidating), but still it fails. And what little I read of it, GeoJSON generally appears to be unforgiving and laden with rigidities.

I've been testing using only regex and a small sample of OpenWeatherMap records wherein I added algebraic signs; i.e., tossed in a couple minus symbols for lon and lat, and added some odd characters, found naturally in the larger dataset.

At this point, I'm not sure what else I can do to get GM to read the OWM stuff. And to be clear, this was more of an educational lark than anything, all stemming from simply wanting to know the OWM city IDs in a given area.

FWIW, I also wrote to [email-masked]<mailto:[email-masked]> asking:

  1.  Why isn't OWM's format the same as the standard GeoJSON?
  2.  And in followup, could you please consider offering the city id list in GeoJSON format in addition to your own OWM format?

Thank you! :)

Kind regards,

Kelly

Their .json file is indeed slightly different than the standard GeoJSON format. I'm not sure whey they chose to get so close to the actual standard but then modify a few things.

Our GeoJSON import capabilities come from a 3rd-party library (GDAL), so we'd need them to add support for this flavor, or you could modify the file directly in a text editor to match the expected format.

I think you could start by changing "coords" to "coordinates", but more may be needed.

Best Regards,

Mike Childs | Global Mapper Guru
Blue Marble Geographics
www.bluemarblegeo.com<http://www.bluemarblegeo.com/?utm_source=signature&utm_medium=email&utm_campaign=website>

22 Carriage Lane
Hallowell, ME 04347 USA
Phone: +1 (207) 622-4622
Toll Free: (800) 616-2725
Email: [email-masked]<mailto:[email-masked]>

Please note, [email-masked]<mailto:[email-masked]> is no longer an active email - please update your tech support contact to [email-masked]<mailto:[email-masked]>.
[Image removed by sender. Blue Marble GeoTalks on March 21,                                  2019]<http://www.bluemarblegeo.com/bmuc/index.php?aid=siggeotalks01152019&utm_source=signature&utm_medium=email&utm_campaign=geotalks_2019&utm_term=http://www.bluemarblegeo.com/bmuc/index.php&utm_content=signature>

[Image removed by sender. Follow us:]

[Image removed by sender. Facebook]<https://www.facebook.com/bluemarble.geographics>

[Image removed by sender. Twitter]<https://twitter.com/BlueMarbleGeo>

[Image removed by sender. LinkedIn]<http://www.linkedin.com/company/blue-marble-geographics>

[Image removed by sender. YouTube]<http://www.youtube.com/user/BlueMarbleWebinars>

[Image removed by sender. Instagram]<https://www.instagram.com/bluemarblegeo/>

The information in this email message is legally privileged and confidential information intended only for the use of the individual(s) named above. If you, the reader of this message, are not the intended recipient, you are hereby notified that you should not further disseminate, distribute, or forward this email. If you have received this email in error, please notify the sender and delete all copies of the message promptly. Thank you.

On Fri, Mar 15, 2019 at 8:38 AM V. Kelly Bellis <[email-masked]<mailto:[email-masked]>> wrote:

OpenWeatherMap.org<https://openweathermap.org/> uses a list of cities by ID: http://bulk.openweathermap.org/sample/city.list.json.gz

A typical record looks similar to the format in GeoJSON, but not similar enough for Global Mapper.

Doubt there's much public need for this, and am just curious - Is it too much bother to expand GM's format choices to include OWM's json? I'll give TextPad and regex  a try just for fun! :)

Thank you! :)

Kind regards,

Kelly

--
V. Kelly Bellis, PLS
17 Union Street
Ellsworth, ME 04605

Precisely Determining True North<https://www.youtube.com/watch?v=zMwDkzl43n8> without a surveyor!
--
V. Kelly Bellis, PLS
17 Union Street
Ellsworth, ME 04605

Precisely Determining True North<https://www.youtube.com/watch?v=zMwDkzl43n8> without a surveyor!

---

From: Erik Wilde <[email-masked]>
Date: Mon, 18 Mar 2019 10:28:49 +0100

hello.

On 2019-03-18 10:06, Martin Daly wrote:
> You have no “FeatureCollection” object, and therefore no “features” 
> array. Your “properties” object does not have commas between consecutive 
> key/value pairs. That’s two problems to be going on with.
> 
> You can use jsonlint.com and geojsonlint.com to check your output, which 
> may give clearer errors than you have been getting.

tl;dr: always pick the right tool for the job.

martin is kind of right, but i think he's not going far enough. these 
JSON problems tell you that it's not such a great idea to edit JSON via 
regexes. to understand why, here's is the greatest answer ever about why 
regex as a tool is limited in its applicability to complex data:

https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454

i'd highly recommend to parse the JSON using a JSON parser of your 
choice (you'll find one in any programming language), do your 
transformation magic on the parse tree, and then serialize the result.

you will find this to be much more robust, extensible, and predictable, 
even though it may seem like a bit of overkill in the beginning.

cheers,

dret.

-- 
erik wilde | mailto:[email-masked] |
            | http://dret.net/netdret    |
            | http://twitter.com/dret    |

---

From: "V. Kelly Bellis" <[email-masked]>
Date: Mon, 18 Mar 2019 08:04:11 -0400

Hi Erik,

That's priceless! Thank you so much for the link 
<https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454>; 
I will treasure it always :)

Sorry my post was too long; I thought I was writing to GeoJASON support; 
not a mailing list.

Mr. Ince's Bruegelesque picture of regex wrestling with HTML aside, as a 
non-programmer and somebody who has never looked at a JSON file before, 
much less a JSON parser, I had no idea what I was walking into when 
thinking that it might be nice to see the OWM city ID list spatially 
displayed in Global Mapper.

I've written to Pavel Zuykov asking if the OWM city ID list could be 
exported as a .kml file from their 'geocoder database' (Pavel's words), 
the same which produced the original .json file

Many thanks to you and to Martin for your kind replies.

Kelly

On 3/18/2019 5:28 AM, Erik Wilde wrote:
>> array. Your “properties” object does not have commas between 
>> consecutive key/value pairs. That’s two problems to be going on with.
>>
>> You can use jsonlint.com and geojsonlint.com to check your output, 
>> which may give clearer errors than you have been getting.
>
> JSON problems tell you that it's not such a great idea to edit JSON 
> via regexes. to understand why, here's is the greatest answer ever 
> about why regex as a tool is limited in its applicability to complex 
> data:
>
> you will find this to be much more robust, extensible, and 
> predictable, even though it may seem like a bit of overkill in the 
> beginning.
>
> cheers,
>
> dret.
>
-- 
V. Kelly Bellis, PLS
17 Union Street
Ellsworth, ME 04605