Map Fixes

Talk about random things WiFiDB related, either new things, concepts, or issues to work out in code.
Post Reply
User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1302
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Map Fixes

Post by ACalcutt »

I've been on vacation the last week so I've made a few long needed fixes to the WifiDB Map.

1.) Fixed many 3 year+ access points not showing. It seems the 3+ year geojson file I was generating had a json syntax error due to a backslash that was in one of the manufactures. in json, backslash is a escape character, so it would hit that access point and just stop, which caused it to ignore about 1 million access points. I am now running all manufactures through json_encode before putting them into the file, so this fixes the syntax error. This brought back many access points I knew were in the database but hadn't been displaying (specifically I was looking for Michigan and Europe, which are now back!).

2.) Tileserver-GL has been updated to v3.0. With that, mapbox-gl.js was also updated to 1.11 which made a longtime issue with my basemap 'water' layer more apparent. I have been stuck at mapbox-gl.js 0.47 for a while since after that version they changed how missing tiles were handled, which caused my incomplete water later to disappear at certain levels. I spent some time updating the style and the layers (see 3) to try and fix the map so I can finally update mapbox-gl, which seems to bring many fixes

3.) Worked on the 'water' layer. In the OpenStreetMap basemap I created last year it seem the ocean layer is messed up after a certain zoom level and it just disappears in certain areas, which made it really hard to tell land from water. This basemap took an extremely long time to generate, so fixing that will take a while, but I was able create a workaround. I was able to download shape files from https://osmdata.openstreetmap.de/ and create a new map layers of land, coatlines, and ocean . Once I had that the new ocean layer I was able to place it on top of the existing water later to fill in the missing gaps in my origional map.

4.) I made some scripts to Import the natural earth shape files (Physical / Cultural) to their own basemap . Not sure what I will use this for yet, but it has all sorts of informational layers that can be used. So far I am using some layers in a style, but unfortunate it doesn't quite get to the detail of the OpenStreetMap basemap.

5.) I made some hardware updates to the OSM generation server. I got a 1TB NVMe disk which is very fast. All this map stuff needs lots of space its constantly doing lots of disk reading and writing, so I think this should help with speed. I also increased the cpu cores and memory to speed up the process a bit more.

6.) I also adjusted tippecannoe so it will now generate APs down to zoom level 18, which it says can get down to 1.5in/4cm detail. This should also mean more visible APs (since tiles get smaller each zoom level, you can fit more detail in them... since I have tiles limited to 500,000 features a tile)
arizonajon
Contributor
Contributor
Posts: 289
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Re: Map Fixes

Post by arizonajon »

Wow, thanks for doing all this, Andrew.

I had noticed some time ago about the apparent lack of my old APs, but have noticed that there's more stuff in the 3+ year category now.

When I tried to come to this website today, Malwarebytes threw a RTP detection Trojan warning. That's new to me. Do you know what that means?

Cheers - Jon N7UV
User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1302
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Re: Map Fixes

Post by ACalcutt »

Hi Jon,

Good to see you are still around. Which site gave you a malware warning? There shouldn't be anything on any of my sites that would cause that, but I have a lot of trouble with AutoIT code getting falsely flagged, so it could be one of my downloads got me on some list again :roll: . Do they give you more detail I could look at?

I'm working from home all the time at the moment, so I've been building a second server here for my map development and generation to stave off boardom. I've also been slowly getting parts to add 10G to my internal network. Sorry if things have been up and down. It seems I am changing direction a bit every week, but I think I am finally getting all the pieces together (though I have a few more 10G network parts on the way). I will say the disk changes I have done so far seemed to improve the wifidb performance quite a bit, so I am happy with that.

In addition to the hardware changes here I have also done a bit of work on the WifiDB frontend. I've re-worked the users page and am trying to work in the messaging I had build a few months back. I've also been adding pagination to more places.

-Andrew
arizonajon
Contributor
Contributor
Posts: 289
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Re: Map Fixes

Post by arizonajon »

Good Saturday, Andrew -

I am not yet an ex-human! (apologies to Monty Python)

Here's the details from my Malwarebytes history:

Malwarebytes
www.malwarebytes.com

-Log Details-
Protection Event Date: 8/26/20
Protection Event Time: 1:53 PM
Log File: 298ff332-e7de-11ea-a3a1-001bdc003d8d.json

-Software Information-
Version: 4.1.2.73
Components Version: 1.0.1003
Update Package Version: 1.0.29095
License: Premium

-System Information-
OS: Windows 7 Service Pack 1
CPU: x64
File System: NTFS
User: System

-Blocked Website Details-
Malicious Website: 1
, C:\Program Files (x86)\Google\Chrome\Application\chrome.exe, Blocked, -1, -1, 0.0.0, ,

-Website Data-
Category: Trojan
Domain: website-frontend.techidiots.net
IP Address: 96.39.84.60
Port: 443
Type: Outbound
File: C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

(end)

I put "techidiots.net" into the allowed list, so now I don't get the warning.

Cheers - Jon
Post Reply