Daemon Generated KML

Talk about random things WiFiDB related, either new things, concepts, or issues to work out in code.
arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Tue Mar 17, 2015 8:54 pm

In addition to the GE problem, over the past few days the Chrome access to WiFiDB wasn't working for the

Files Waiting for Import
Files Already Imported
Daemon Generated KMZ

pages. The page would "stutter" every second or so and never load or finish loading. Had to switch to Firefox to get access again.

So, I decided that I'd try wiping all my browser data for the past week, as less than a week ago Chrome was fine with the website. Did that and that solved the problem. It might have been some odd browser setting or temp info that got stuck in there and made a mess.

Emboldened, I thought maybe I have the same issue with GE. So, I checked a few sites on GE slow to load problem, and it seemed like there too clearing the cache might be the easy way to fix the problem. Logged out of the GE server, cleared the cache, closed GE, restarted, and the link to the full DB now works fine.

Sorry for the operator error. Now I can see the data again! Memory usage when running the full DB link is now only 2250 MB, down from 3700+ before. Also, I can actually zoom out to see all of Arizona (takes forever to load, though). CPU usage for GE is 13%.

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Mon Mar 23, 2015 3:20 pm

I thought that the issue with GE crashing had been resolved through the cache emptying. Alas, it once again cannot survive having the WiFiDB Full export button selected. Memory usage quickly climbs to 3700+ MB and then shortly later it crashes. I've seen it survive memory usage of up to 3300 MB, then drop back to 2500 MB, but when it gets to the 3700+ number it is doomed.
I did exactly as you suggested; I use the WiFiDB Newest AP first, so I'm only 1000 m above the terrain, and in any event shouldn't see more than several hundred simultaneous APs. Then I click the Full Export no label button and within 30-60 sec GE becomes unresponsive and then crashes. I've moved GE to isolated locations where there are no APs recorded and zoomed in to levels where I'm looking at a block or two, but the crash occurs anyway. I've increased the memory and disk caches to their maximums.
GE has no issues with either the WiFiDB Daily Export (label or non-label) or the Newest AP.

Here are the details on the GE version.

Google Earth
7.1.2.2041
Build Date
10/7/2013
Build Time
12:28:36 pm
Renderer
DirectX
Operating System
Microsoft Windows (6.1.7601.1)
Video Driver
Google Inc. (00010.00018.00010.03960)
Max Texture Size
8192x8192
available video memory
1760 MB
Server
kh.google.com

As an alternative to the network link for the full DB, I downloaded the newest full DB labeled file. I then started GE, cleared cache, made sure no network links were enabled, then attempted to open the DB file in GE. It took many minutes (5-10?) to just load into GE, with GE frozen the whole time. Once it unfroze, I noted that the radio button to enable the file was not checked, so I tried to explore the tree for the WiFiDB Full Database Export. I see that there's a Full Database Export subentry, and then within that a "WifiDB Newest AP" entry, and a "Regions to save precious CPU cycles." entry (there's nothing in it), then a very long list of individual APs listed below. I can scroll up/down to see all the 400k APs.

Now, I click the radio button to enable the entire DB file. GE freezes. It takes 10-15 minutes before GE unfreezes. When it finally starts to respond again, in the PHX area none of the APs newer than about 2010 are displayed on my screen. However, I can page down through the tree list and select an individual AP, and discover that indeed many of my APs are there and will be displayed if I select them directly. Also, this works for APs from other contributors, but obviously it takes a looooooong time (5 - 10 min) to pan from AZ to Germany. However, GE doesn't crash.
All this time the memory consumption for GE is creeping up bit by bit (well, byte by byte) but it's still under 3000 MB. GE responds quickly to move to another AP which is nearby the current view, and if an AP is selected which requires a significant move (many miles or more) then GE slows down as expected. But it doesn't crash. For instance, moving from one AP to another in Germany takes only a few seconds (the built-in GE move process); moving from an AP in Germany back to an AP in Arizona takes 5 minutes but will intermittently render whole portions of the globe during the pan if there's not too many APs to be displayed, which I would expect. However, when it finally gets out of the way of MA, then it quickly zooms into the AP position in AZ. However, for most of the APs it will not show an icon, only the text box.
The attachment Image 2.png is no longer available
I just found (went back in the list to look for an AP captured back in 2010 in AZ) that all the APs captured back then do render on the screen.
Image 3.png
Image 3.png (1000.98 KiB) Viewed 5375 times
And GE is quick to pan back and forth.

It's almost like GE has run out of the ability to render the icons later than a certain date, which would be also after a certain count. Maybe breaking the file/network link into subdirectories by year or month could be an easy initial way to reduce the overall burden on GE?
Of course, if no one else is suffering from this problem, then it's something about my computer/setup...

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Mon Mar 23, 2015 3:21 pm

Image 2.png
Image 2.png (1.09 MiB) Viewed 5375 times
don't know why image2.png reported as no longer available. here it is

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Mon Mar 23, 2015 4:49 pm

Jon,

I am able to load the full kml here and it doesn't seem to be missing any icons that I can see.

You may want to try exporting just your access points using the Export Page ( https://live.wifidb.net/wifidb/opt/expo ... func=index ) or your all ap page (https://live.wifidb.net/wifidb/opt/user ... arizonajon). Since this export just has your exports...it will cut out about 250,000 access points. *note it may take the wifidb ~10 minutes to export your full file since there are so many APs*


One thing I notice the last time you had the issue your mentioning is phil had made a change to the kml export that added a lot of extra white lines. this had made the full export much bigger and raised the kmz size to 17.41MB . After reverting that change the kmz size went down about 1MB and it started working for you again. However with the access points you have added the kmz size is up to 18.37MB. it is also important to note a KMZ file is basically a zipped KML. The latest KMZ is 18.37MB, but the actual fulldb KML is a 452MB file (you can see this yourself by changing the KMZ extension to .zip and looking inside the file). 452MBs of kml is a huge amount of xml that google earth needs to load into memory, so its not a big supprise google earth has trouble with it.

Like Phil and I mentioned we are working on a new kml format, but it may take some time to get a working solution. We may be asking you to do some tests of this new format once we think we have something that will work. I think the plan right now is to break the fulldb down into smaller files (most likely by each imported list) and put the regions around each list so it only shows whats in view.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Mon Mar 23, 2015 4:55 pm

Also, the issue with the forum images should be fixed. I had disabled a sync process because I though it wasn't in use anymore...turns out it was. should be fixed now.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Tue Mar 24, 2015 11:22 am

I've tried exporting only my APs, but even after 15-30 minutes nothing happens. In fact, the page goes blank after about 15 minutes. Perhaps it takes a very long time? On the other hand, I've tried exporting other users' stuff, and it takes only a few moments. Even Andrew's file is only a 6 minute process...

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Tue Mar 24, 2015 11:33 am

That is the PHP.INI file setting the max execution time for a script to run on apache. Not sure what Andrew has it set to. I was also thinking about moving full user exports to the export daemon to keep them up to date and not be ad hoc. Andrews user does not have that many APs since most are in the recovery user now. So that would make sense as to why his were able to be exported.
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Tue Mar 24, 2015 1:08 pm

pferland wrote:That is the PHP.INI file setting the max execution time for a script to run on apache. Not sure what Andrew has it set to. I was also thinking about moving full user exports to the export daemon to keep them up to date and not be ad hoc. Andrews user does not have that many APs since most are in the recovery user now. So that would make sense as to why his were able to be exported.
I doubt its max execution time in php.ini...it is set to 3600 seconds (60 minutes)

I exported Jons file about a week ago and it worked, but took about 10 minutes (it won't show anything when generating). I'm gunna give it a try and see if its working.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Tue Mar 24, 2015 1:20 pm

Looks like this is the issue. Out of allowed memory
[Tue Mar 24 18:14:38 2015] [error] [client 209.239.201.208] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 205337736 bytes) in /srv/www/virtual/live.wifidb.net/wifidb/lib/createKML.inc.php on line 669, referer: https://live.wifidb.net/wifidb/opt/expo ... func=index

I wonder if this is still using the old style of building the list that used an array, which used more memory. I'll have to look at that.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Tue Mar 24, 2015 9:35 pm

I've been messing with an api to dynamically gernerate kmz files. I am still in early stages with this, but it is one way I am looking to get around the memory issue.

I made a few test functions

One to export a list based on the list ID
https://live.wifidb.net/wifidb/api/expo ... st&row=315

One to export users lists
https://live.wifidb.net/wifidb/api/expo ... arizonajon

One to export all lists, seperated by user
https://live.wifidb.net/wifidb/api/expo ... ll_netlink

One to export a Combined KMZ
https://live.wifidb.net/wifidb/api/expo ... ed_netlink


These are still in early stages...so expect them to break or change

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Sat Jul 04, 2015 2:13 pm

Hi guys -

It's been a while - I got a new job in April and have been getting involved in that.

However, a man's gotta drive, so I continue to collect APs. I'm up to over 54% of the overall database size. While that's fine from a statistics PoV, i have been suffering mightily from GE being unable to render properly (at least that's what I'm thinking it is).

I know that the following may be only my problem, since I have recorded so many APs and so many runs. It's likely GE is running into significant issues.

Here's what I've been seeing. First, clear all GE caches. When I start my network link, or refresh it, GE starts populating with APs as it successfully fetches each run from the server. However, as it continues to fetch runs, other runs vanish from the display. The run file remains on the sidebar, but the APs no longer display. If I try to refresh the run, it still won't display.

Am I just doomed due to GE being unable to render?

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Sat Jul 04, 2015 2:14 pm

On perhaps a slightly different topic, one run which to my knowledge has never displayed is the one with ID 4732 and file name 49507292_2015-06-27_13-52-37_AutoSave.VS1. This is part of a tour I was making one day, heading out Bethany Home Rd to its end then coming back on Indian School. On the GE display, the other parts of the tour show up, but this particular file, while listed in the list of already imported runs/files, it appears to be empty even though the summary shows 1000/487 and a file size of 247.32 kB.

I've tried to resubmit this file to the server, but it apparently rejects it because the file name is same. So, before I tweak the file name I was wondering if one of you could take a look at the file submitted and see if there's something weird about it that prevents it from showing any APs.

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Sun Jul 05, 2015 9:01 am

When I look at that file, 4732, it seems like it imported properly and I am able to export the kmz ok.

https://live.wifidb.net/wifidb/opt/user ... t&row=4732


EDIT: I also noticed the link for 4732 is pointing to id 4738. 4738 is giving me the blank page you mentioned. I forget why this ID is different...it may be because this is supposed to be a user list, but i'm going to have to look into it

https://live.wifidb.net/wifidb/opt/user ... t&row=4738


Can you tell me if the one that works ( https://live.wifidb.net/wifidb/opt/user ... t&row=4732 ) is the list you are missing?

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Sun Jul 05, 2015 10:38 pm

The last link you provided (to 4732) is from 200205 to 201642; this is already available via the arizonajon network link. That file is two datasets before the one I'm having troubles with.

I've attached a copy of the file "2015-06-27 13-52-37_AutoSave.VS1". It's timestamped from 203628 to 205237.
Attachments
2015-06-27 13-52-37_AutoSave.VS1
(247.32 KiB) Downloaded 96 times

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Tue Jul 07, 2015 7:18 am

Jon,

For your import issue I am thinking it is time to put some fixes phil has made in his beta branch of wifidb in my instance of wifidb. This updates includes improvements the import code I think may help.

Unfortunately due to some changes in the sql database I am going to have to do a rebuild of the wifidb data, which means we will export all the file and user information and rebuild the database back from the original VS1 files. In this process all file information (filename, notes, import user) will be transfered over while the AP data will get rebuilt from the vs1 file.

Right now I have merged phils branch into mine here ( https://github.com/pferland/WiFiDB/tree/andrew-test-dev ) and I am working on a few bugs before putting it into production.

When we are ready to upgrade we will disable imports, export all file import information, export user information, export the original VS1s and then I will rebuild with the new version of wifidb.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Thu Jul 09, 2015 5:40 pm

Jon,

I was looking for google earth alternatives and imported your network link into "ArcGIS Explorer". It took a while to load, but it seem to be able to browse the access points a little better that google earth (though it still struggles a little).


Your network link was pretty cool looking (see the attached screenshot). Its really amazing how much you uploaded and how much time all of it must have taken.

BTW, congrats on the new job!
Attachments
arizonajon_arcgis_explorer.png
arizonajon_arcgis_explorer.png (6.85 MiB) Viewed 5095 times

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Mon Sep 28, 2015 2:46 pm

Holy shit Jon,

Your AP count is getting freaking huge! Getting us close to the elusive 1 million AP count. I never thought the Database would ever get this big when we started out with about 10,000 APs.

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Fri Oct 02, 2015 12:47 pm

That image above is a thing of beauty! :shock:

Or perhaps horror realizing that I purposely most of the X-Y gridded streets just to collect APs. I gotta get a life! However, there are breweries in some far-flung locations here in the Valley so sometimes I can set a goal...

Thanks again to both of you for the work you've done to make this a pretty darned smooth and troublefree collection system. The upload feature has further simplified the work, so when I pull into the neighborhood and connect to the rooftop hotspot, I can start my upload quickly.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Sun Oct 18, 2015 10:43 am

I spent some time last week working on the daemom kmz and the api dynamic kmz

The Daemon Full KMZ now creates a file with a structure more like the api. Instead of all access points dumped into one large list there is now a structure based on usernames and lists. Each list is now a separate kml file that gets stored inside a main KMZ file (if you look inside the full kmz as a zip file you will see a files folder with every list in its own pre-made kml file). I have also made some progress with regions, so this helps hide stuff you arn't looking at and improves performance of google earth when looking at the full database (it will still have trouble if there are a lot of points on the screen...but it still helps a lot). I have also made changes to improve memory use during the daemon kmz generation which was causing the daemon crash and take forever updating. The benefit to using the daemon full kmz is all kml files are created in advance, so google earth doesn't have to go to the server to get each list (they are already included inside the kmz file)

The dynamic api has also been updated to support the regions I made for the daemon kmz. Each list now has a region around it, so it doesn't load unless you are looking at the region. This helps with load on the server and should help with google earth performance

There are still a few cases where the regions are hiding things that should be in view. I am still working on those issues.

Post Reply